Moving to the US

I’m moving to the US, and as part of that I’m wrapping up a host of legal things. I have some of this documented for my own benefit, and some friends are talking about similar things, so I decided to just write it all down.

Disclaimer first: I am not a lawyer, do not take this as legal advice, your mileage may vary, I’m talking about my position, not yours, etc etc


As a Canadian citizen, I’ll be entering the US on a TN visa. TN is granted at the border, or you can mail in the I-129 form.

I renewed my passport early (more than 12 months before it would normally expire), citing the 3 year period of the TN visa as the reason in the passport application.

My company is handling the TN visa application, I’ll know more once the paperwork goes through.

Taxes in Canada

The CRA has an excellent section on emigrating from Canada.

If you want to be entirely through, the CRA has a form to determine what you should file as – emigrant/deemed resident. I don’t meet any of the deemed resident requirements, so I’ll be considered an emigrant.

I will be filing Form T-1161 along with my tax return for 2017, because becoming an emigrant means that my property undergoes a deemed disposition. Essentially, the CRA will consider all my property to be sold and repurchased at market value, so any capital gains can be taxed.

Note: One of my friends brought up that Canadian deemed residency includes a “183 day rule”, which says that you’re a resident of Canada if you’ve been in Canada for 183 days or more in a calendar year. He was concerned because it could mean that you file as a tax resident of both Canada and the US if you start your job late enough.

In my (see disclaimer) reading of it, it only matters if you’re either entering Canada, or visiting Canada. Since you’re leaving Canada, you’re going to be taxed on all your income prior to leaving Canada, and any income from Canada after you leave. Essentially, the 183 day rule doesn’t apply in this case.

If you want to be perfectly clear on your tax status, ask the CRA what they think of your situation by filing Form NR73.

Taxes in the US

I’m probably going to pay someone to do my taxes, at least for the first year when I move. I believe I would be considered a Dual Status Resident Alien.

The IRS does not like TFSAs (or RESPs, but recent grads are unlikely to have those). I’ve moved everything that I had in a TFSA into a RRSP, which the IRS likes a lot better.

Keeping my Canadian accounts means that additional forms (I know of Form 8938 and the FBAR form with the US Treasury) and will need to be filed for future tax years, which is always fun.

As a side note, California taxes Canadian RRSPs, so it’s a good thing I’m going to Seattle.

No Comments

Exploring AWS Organizations

When AWS Organizations went GA, I was really happy. While I’ve had my personal AWS account for a while, I have a bunch of sites that aren’t personal in nature, and I wanted to spin them off into another account with the intention of having enough isolation that I could apply Terraform to those accounts freely.

The Good Parts

Inviting an existing account was painless

I performed this with the other account open in an incognito session. It was a simple matter to send the invitation from one account, and the offer showed up in under a minute in the Organizations tab in the other account.

The entire process took under 5 minutes.

Cross Account Access was set up automatically

I’ve spent a bunch of time setting up role switching. I really appreciate the automatic setup. (Though see Nit #2.)

Creating accounts is super simple

It’s 3 fields on the UI, only 2 are compulsory. The vast majority of information on the billing panel (Address, etc) is automatically copied from the root account.

SCPs are powerful

I blocked Route 53 as an experiment. After I applied the SCP, I can’t do anything in the R53 console:

Update: I had a nit about SCP policy inheritance being confusing – it was actually a misunderstanding about how policies are applied. I applied a restrictive policy to the root OU, expecting that it would result in default deny for accounts, but attaching a separate policy would override the restrictive policy. Unfortunately, the restrictive policy blocked me from being able to do anything, even on accounts with a FullAWSAcess policy attached.

This is a security feature – child nodes can’t be allowed to set a more permissive policy than what a (grand)parent specifies. So even if the child has the FullAWSAccess policy applied (like I had), it’s ineffective because my root said “Only have access to read-only billing details”, and the intersection of allowed permissions was “read-only access to billing data”.

As such, the root policy that should be something that either allows or denies something that you want applied to all accounts in your organization (except the master account). In practice I think people are going to end up leaving the FullAWSAccess policy on the Root OU, and apply more selective policies to nodes.

The Less Good Parts/Nits

The process wasn’t as pain free as hoped, here’s some rough edges I found, pretty much all issues I found on first use, and avoided once I knew about them:

1. The progress of account creation through the console isn’t clear.

Specifically, after clicking the “Create” button on the Add Account page, I get redirected back to the account list, and there’s a new row that just says “Creating…”.

There doesn’t appear to be an automatic refresh, so I manually refreshed. Now the “creating” message has disappeared, and a new account with an empty name and an account number appears in the account list.

Refresh a few more times, and the name eventually appears. I’m guessing the name field is only populated after the account is set up on the backend.

AWS: It’d be nice if the newly created account was somehow marked as “creation in progress”.

2. It is unclear how to get access to the newly created account.

AWS Orgs is probably aimed at a more experienced subset of AWS users, so I’d imagine they’d know how to do this. But it would be nice to explicitly mention that you have to role-switch to the named role that was created as part of the account. Something like “To access the new account, you need to switch role from your master account, using the account number and the name of the IAM role you provided when the account was created.”

You (also) don’t appear to be able to role switch to it as soon as the account number is listed in the account list. You have to wait until the account name shows up in the account list before you can role switch to the new account.

AWS: Also, the flow of having to go out of the Org page back to the AWS console to set up the switching seems easily optimized – a “Switch Role” button next to each account would be helpful.

3. What’s with all the emails?

I’m unsure if this will be fixed – right now it appears that Orgs is hooking onto the normal creation process. This means for every account you create you get “Welcome to Amazon Web Services” and “Your AWS Account is Ready – Get Started Now” emails sent to the new account’s email address.

4. The Auto-Created Role

First off – The role created by Organizations uses an inline AdministratorAccess policy, not the AWS managed one. Considering it has the same name (and actual Policy details) as the managed policy, I’m not sure why an inline policy was used. For what it’s worth, I tried swapping it out on my test account, and there were no adverse effects.

Secondly, what happens if I create an account, but don’t specify a role name? I’m guessing an account gets created without the cross account access setup, and I have to do the forgot-password dance on the email address I created the account with to get access. I’ve verified that you can do the forget password dance to set a password for the root account..

5. You can’t delete an account

I jumped in head first and created a (badly named) testing account before realizing this: Deleting accounts is (currently) not possible. The feature might be coming though! The “Remove Account” button is misleading – it will happily allow you to attempt to remove a created account, only for you to get an error.

The Organizations backend clearly knows which account is a created account (the status column shows “Created” vs “Joined”), so the “Remove Account” button could be disabled when a created account is selected. This is already done for the Master Account, so it might be simple to extend to created accounts. Or relabel the button “Remove Invited Account” to be super clear.

As a side note, I might be able to close a created account through the Billing panel (there’s a “Close Account” checkbox), but I’m not sure what would happen to my master account – and I’m kind of attached to it, so I’m not going to risk it.

Related, I can’t delete an account that failed to be created.

I managed to do this accidentally – see nit #1. I thought the initial creation had failed, and I tried to create the account again. The second request got accepted by Organizations, only to eventually error out because I reused the account email address.

6. A grab bag of other stuff

Three small things I encountered that didn’t really deserve their own section.

  • Orgs is keeping (caching?) their own copy of the Account Name. I renamed an account in the AWS Console, but the change didn’t replicate back to the Org account list. So there might be name drift if you rename accounts (there appears to be no way to rename an account through Organizations.)
  • It is unclear what happens if I update the details in the master account (eg the address). I’m guessing it’s not replicated to the other accounts.
  • The default VPC in my master account is The default VPC in a created account is – but for every created account. I’m not sure if this is accidental or intended, but the result is I can peer the default VPC in my root account with a default VPC in one of the created accounts. If it is intended, would it be possible to look at current VPCs and select a CIDR that isn’t in use?

7. SCPs are confusing

My original point here was the result of misunderstanding how the policies are applied. Only thing I’d suggest is add an “Effective Policy” listing under the SCP entry for each node.

SCP wasn’t enabled by default, despite my selecting Enable All Features in Settings. There’s a second step I missed: you have to enable SCPs on the root OU, not the master account – something which is documented, but I completely missed.

Tip: After navigating to the “Organize Accounts” tab, click “Home” to see the top level OU. The list of your acccounts and OUs is not the top level of your organization.

Also, having an SCP of FullAWSAccess appears to be required on the root OU. I removed it (after creating a stub policy, because at least one policy is required to be attached to the root), and promptly lost the ability to do anything in a cross-account role, even though those accounts still had the FullAWSAccess policy attached to each account individually:

To be clear, default allow is the sane default – the intention here is probably “You should be putting accounts in OUs and black/whitelisting on that”. But being unable to use role switching after the SCP on the root OU was weird.

I’m assuming there’s a combination of resources and permissions that I can add to an SCP that would allow the cross account stuff and nothing more. It’d be nice if AWS provided this as a reference.

That said

It could have gone far worse. The invite process was smooth for existing accounts, and Orgs is doing what is says it will, with some exceptions on the UI side of things.

I’m very sure they’ll iterate and improve it. I’m personally hoping there’s going to be a way to move resources between accounts. Being able to move S3 buckets to another account alone would be amazing.


1 Comment

Terraform import with AWS profiles other than default

I’ll come back and clean this up, but for now:

Undocumented: It will use the default AWS profile – it will pull in your shared credientials, and use the default values if specified.

As per code, use AWS_PROFILE=<name> terraform import aws_db_instance.default <id> to import using a AWS profile that isn’t default.

ELB holds onto subnets that are to be destroyed. Combined with new subnet having the same CIDR, can’t create new subnet because CIDR is in use, can’t update ELB because new subnet isn’t created yet.

No Comments

Printing the table structure of a SQLite Database in Android

I’m doing some Android app development, and as part of it, I hit some issues with the database.

My first plan was to download the database and open it in SQLite, but having to re-establish my ADB debug session each time I downoaded the file got annoying, so I decided to write a short snippet which dumps the name & CREATE TABLE schema of each table to the log:

SQLiteDatabase db = YourDbHelper.getWritableDatabase();
Log.d(this.getClass().getSimpleName(), db.getPath());
Cursor c = db.rawQuery("SELECT type, name, sql, tbl_name FROM sqlite_master", new String[]{});
while (!c.isAfterLast()){
        int count = c.getColumnCount();
        Log.d(this.getClass().getSimpleName(), Integer.toString(count));
        Log.d(this.getClass().getSimpleName(), c.getColumnNames().toString());
        for (int i = 0; i < count; i++){
                Log.d(this.getClass().getSimpleName(), c.getColumnName(i));
                Log.d(this.getClass().getSimpleName(), c.getString(i));

This gives decently printed output for a quick and dirty script:

07-16 23:44:53.795/ D/ProjectListActivity: name
07-16 23:44:53.795/ D/ProjectListActivity: projects
07-16 23:44:53.795/ D/ProjectListActivity: sql
07-16 23:44:53.795/ D/ProjectListActivity: CREATE TABLE `projects` ( `_id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, `name` TEXT, `avatar` TEXT, `short_description` TEXT, `description` TEXT)

No Comments

Quick and Dirty Shoestring Startup Infra

At the University of Waterloo, we have a Final Year Design Project/Capstone project. My group is working on a conference management suite called Calligre. We’ve been approaching it as kind of a startup – we presented a pitch at a competition and won! While sorting out admin details with the judges after, they were oddly impressed that we had email forwarding for all the group members at our domain. Apparently it’s pretty unique.

In the interest of documenting everything I did both for myself, and other people to refer to, I decided to write down everything that I did.

Note that we’re students, so we get a bunch of discounts, most notably the Github Student Pack. If you’re a student, go get it.


  1. Purchase a domain. NameSilo is my go-to domain purchaser because they have free WHOIS protection, and some of the cheapest prices I’ve seen.
    Alternatives to NameSilo include NameCheap and Gandi. Of the two, I prefer Gandi, since they don’t have weird fees, but Namecheap periodically has really good promos on that drop a price significantly.
  2. Use a proper DNS server. Sign up for the CloudFlare free plan – not on your personal account, but a completely new one. CloudFlare doesn’t have account sharing for the free plan yet, so Kevin and I are just using LastPass to securely share the password to the account. For bonus points, hook CloudFlare up to Terraform and use Terraform to manage DNS settings.
    Alternatives include DNSimple (2 years free in the GitHub Student Pack!) and AWS Route 53.
  3. Sign up for Mailgun – they allow you to send 10000 messages/month for free. However, if you sign up with a partner (eg Google), they’ll bump that limit for you. We’re sitting at 30000 emails/month, though we needed to provide a credit card to verify that we were real people.
    Follow the setup instructions to verify your domain, but also follow the instructions to recieve inbound email. This allows you to setup routes.
    Alternatives include Mailjet (25k/month through Google), and SendGrid (15k emails/month through the Github Student Pack) – though SendGrid doesn’t appear to do email forwarding, they will happily take incoming emails and post them to a webhook
  4. Once you have domains verified and email setup, activate email forwarding. Mailgun calls this “Routes”. We created an address for each member of the team, as well as contact/admin aliases that forward to people as appropriate. I recommend keeping this list short – you’ll be managing it manually.


  1. We currently have a basic landing page. This is still in active development, so we use GitHub Pages in conjunction with a custom domain setup until it’s done. This will eventually be moved to S3/another static site host. For now though, it’s free.
  2. Sign up for a new AWS account.
  3. Register for AWS Educate using the link in the Github Student Pack. This gets you $50 worth of credit (base $35 + extra $15). Good news for uWaterloo people: I’ve asked to get uWaterloo added as a member institution to AWS Educate, so we should be getting an additional $65 of credit when that happens.
    Note that if you just need straight hosting, sign up for Digital Ocean as well – Student Pack has a code for $50/$100 of credit!


  1. In AWS, create an IAM User Account for each user. I also recommend using groups to assign permissions to users, so you don’t need to duplicate permissions for every single user. I just gave everyone in our team full admin access, we’ll see if that’s still a good idea 6 months down the road…
  2. Change the org alias so people don’t need to know the 12 digit account ID to login.
  3. Enable IAM access to the billing information, so you don’t have to flip between the root account & your personal IAM account whenever you want to check how much you’ve spent.
  4. Enable 2 factor auth on the root account at the very least. Let another person in the team add the account to their Google Authenticator/whatever, so you’re not screwed if you have your phone stolen/otherwise lose it.

More stuff as and when I think about it.


No Comments

Notes from various AWS Investigations

  • AWS CloudWatch Logs storage charge == S3 storage charge. Possibly less, since the logs are gziped level 6 first.
  • CW Logs makes more sense than using AWS Elasticsearch at small scale – prices start at 1.8c an hour + EBS charges vs 50c/GB of log ingestion + storage
  • For pure log storage & bulk retrival, S3 makes far more sense than either ElasticSearch or CloudWatch Logs. B2 is ~20% of S3 though, so they make even more sense.

  • DynamoDB streams are for watching what happens to a table, and they rotate every ~24 hours, so you’d get charged on a rolling basis, and can’t delete individual events. I’m assuming events don’t disappear once you’ve processed them.

  • Cert Manager is in more zones! But only makes a difference if you hang stuff in front of an ELB. Certs for CloudFront have to still go through us-east-1.
  • API Gateway has direct integration with DynamoDB, doing an end run around Lambda functions that just insert & retrieve records ( Amusingly, models continue to not be used. (I still don’t understand what Models are supposed to do/enforce)
  • DynamoDB cross-region replication is weird. You spin up an EC2 instance that handles it for you. I wonder if the DynamoDB team will work on managed replication…
  • DynamoDB is stupid cheap, and it makes sense for me to migrate the vast majority of my DB centric stuff to it.
  • CloudFront has a weird “$0.000 per request – HTTP or HTTPS under the global monthly free tier” for requests, and I’m not sure why. My account is long out of the free tier.

No Comments

Improving my OpenVPN Ansible Playbook

I had a working OpenVPN configuration. But it wasn’t the best it could be. The manpage for OpenVPN 2.3 ( was used to find particularly interesting options.

For most of the changes I had to find examples and more information through Googling, though is of particular note for popping up very often.

Improving TLS Security

  1. Added auth SHA256 so MACs on the individual packets are done with SHA256 instead of SHA1.

  2. Added tls-version-min 1.2 to drop SSL3 + TLS v1.0 support. This breaks older clients (2.3.2+), but those updated versions have been out for a while.

  3. Restricted the tls-ciphers allowed to a subset of Mozilla’s modern cipher list + DHE for older clients. ECDSA support is included for when ECDSA keys can be used. I’m uncertain of the usefulness of the ECDHE ciphers, as both my client and server support it, but the RSA cipher that’s 3rd in the list is still used. Continuing to investigate this.

The last 2 changes are gated by the openvpn_use_modern_tls variable, which defaults to true.

  1. New keys are 2048 bit by default, downgraded from 4096 bit. This is based on Mozilla’s SSL guidance, combined with the expectation of being able to use ECDSA keys in a later revision of this playbook.

  2. As part of the move to 2048 bit keys, the 4096 bit DH parameters are no longer distributed. It was originally distributed since generating it took ~75 minutes, but the new 2048 bit parameters take considerably less time.

Adding Cert Validations

OpenVPN has at least two kinds of certification validation available: (Extended) Key Usage checks, and certificate content validation.


Previously only the client was verifying that the server cert had the correct usage, now the verification is bi-directional.
OpenVPN, more about EKU checks: 1 & 2

Certificate content

Added the ability to verify the common name that is part of each certificate. This required changing the common names that each certificate is generated with, which means that the ability to wipe out the existing keys was added as well.

The server verifies client names by looking at the common name prefix using verify-x509-name ... name-prefix, while the client checks the exact name provided by the server.

Again, both these changes are gated by a variable (openvpn_verify_cn). Because this requires rather large client changes, it is off by default.

Wiping out & reinstalling

Added the ability to wipe out & reinstall OpenVPN. Currently it leaves firewall rules behind, but other than that everything is removed.

Use ansible-playbook -v openvpn.yml --extra-vars="openvpn_uninstall=true" --tags uninstall to just run the uninstall portion.

Connect over IPv6

Previously, you had to explicitly use udp6 or tcp6 to use IPv6. OpenVPN isn’t dual stacked if you use plain udp/tcp, which results in being unable to connect to the OpenVPN server if it has an AAAA record, on your device has a functional IPv6 connection, since the client will choose which stack to use if you just use plain udp/tcp.

Since this playbook is only on Linux, which supports IPv4 connections on IPv6 sockets, the server config is now IPv6 by default (, by means of using {{openvpn_proto}}6.

Hat tip to T-Mobile for revealing this problem with my setup.


  1. Add revoked cert check

  2. Generate ellptic curve keys instead of RSA keys However, as noted above, ECDHE ciphers don’t appear to be supported, so I’m not sure of OpenVPN will support EC keys.

  3. Add IPv6 within tunnel support (Possibly waiting for OpenVPN 2.4.0, since major changes are happening there)

This SO question seems to be my exact situation.

Both this SO question and another source are possibly related as well.

Tried splitting the assigned /64 subnet with:

ip -6 addr del 2607:5600:ae:ae::42/64 dev venet0
ip -6 addr add 2607:5600:ae:ae::42/65 dev venet0
  1. Investigate using openssl ca instead of openssl x509next version of easyrsa uses ca

, ,

No Comments

Using Amazon S3 + CloudFront + Certificate Manager to get seamless static HTTPS support

TL;DR: This post documents the process I took to get S3 to return redirect requests over HTTP + HTTPS to a given domain.

I’m trying to trim down the number of domains and subdomains that I host on my server, since I’m trying a new policy of moving servers every few months in an attempt to make sure I automate everything I can.

One of the things that I’ve done was to start consolidating static files under a single subdomain, and use 301 redirects in nginx to point to the new location. Thanks to Ansible, rolling out the redirect config is a matter of adding a new domain + target pair to a .yml file and running it against a server.

But it’d still be nice if I didn’t have as many moving parts – which meant that I looked at ways to get an external provider to host this for me. I decided to try getting S3’s static website hosting a try to see if it supported everything I wanted it to do. In this case, I want to return either a redirect to a fixed URL, or a redirect to a different domain, but same filename. Essentially, my nginx redirect configs are either return 301$request_uri or return 301

For the purposes of this post, let’s assume I have the domain that redirects to my Twitter profile.

Creating the S3 Bucket

I knew that S3 could do static site hosting – but the docs seem to indicate that while redirecting to a different domain is possible, it will still use the same path to the file. So this would work for the different domain, same file name case, but not the fixed URL case.

I found my solution in an example of redirecting on an HTTP 404 error – but it could be adjusted to redirect all the time by removing the Condition elements.

So let’s create the bucket with the AWS CLI: aws s3api create-bucket --bucket tw-kyle-io --create-bucket-configuration LocationConstraint=us-west-2

Then I had to apply the redirection rules. The CLI uses JSON format to specify the redirect rules, so to make things simple, I dumped the config into a file:

  "IndexDocument": {
    "Suffix": "redirect.html"
  "RoutingRules": [
      "Redirect": {
        "HostName": "",
        "Protocol": "https",
        "ReplaceKeyWith": "lightweavr"

and then called it through the CLI: aws s3api put-bucket-website --bucket tw-kyle-io --website-configuration file://website.json. Note that the use of IndexDocument is misleading: I never actually created a file in S3, but the S3 API requires that something be specified for that key.

People who have created a static website in S3 might be saying that I used a bad bucket name, because now I can’t use CNAMEs with the bucket. Well, I have the useful experience of writing this after discovering that HTTPS requests to the bucket don’t work because S3 doesn’t support HTTPS to the website endpoint. CloudFront doesn’t have restrictions on bucket naming, especially where we’re going to use the website endpoint so we can use redirections. (Hat tip to a StackOverflow question and another one as well.)

Amazon Certificate Manager

So I ended up using CloudFront. But first, I had to create a certificate with ACM. Because I’m creating a subdomain cert, I had to use the CLI – the console only allows you to create a * or cert.

aws acm request-certificate --domain-name --domain-validation-options,

I had to approve the certificate creation, which required waiting for the email. If you try to create you get an error about the cert not existing: A client error (InvalidViewerCertificate) occurred when calling the CreateDistribution operation: The specified SSL certificate doesn't exist, isn't valid, or doesn't include a valid certificate chain.

CloudFront Magic

Then, it was time to configure the CloudFront distribution. Which isn’t trivial, there’s a bunch of options, and it wouldn’t surprise me if I screwed something up. To get the options below, I created a distribution through the AWS console, then compared that against a generated template (aws cloudfront create-distribution --generate-cli-skeleton).

I put the following in a file named cf.json

    "DistributionConfig": {
        "CallerReference": "tw-kyle-io-20160402",
        "Aliases": {
            "Quantity": 1,
            "Items": [
        "DefaultRootObject": "",
        "Origins": {
            "Quantity": 1,
            "Items": [
                    "DomainName": "",
                    "Id": "",
                    "CustomOriginConfig": {
                        "HTTPPort": 80,
                        "HTTPSPort": 443,
                        "OriginProtocolPolicy": "http-only",
                        "OriginSslProtocols": {
                            "Quantity": 1,
                            "Items": [
        "DefaultCacheBehavior": {
            "TargetOriginId": "",
            "ForwardedValues": {
                "QueryString": false,
                "Cookies": {
                    "Forward": "none"
                "Headers": {
                    "Quantity": 0
            "TrustedSigners": {
                "Enabled": false,
                "Quantity": 0
            "ViewerProtocolPolicy": "allow-all",
            "MinTTL": 86400,
            "AllowedMethods": {
                "Quantity": 2,
                "Items": [
                "CachedMethods": {
                    "Quantity": 2,
                    "Items": [
            "SmoothStreaming": false,
            "DefaultTTL": 86400,
            "MaxTTL": 31536000,
            "Compress": true
        "Comment": "Redirect for",
        "Logging": {
            "Enabled": false,
            "IncludeCookies": false,
            "Bucket": "",
            "Prefix": ""
        "PriceClass": "PriceClass_100",
        "Enabled": true,
        "ViewerCertificate": {
            "ACMCertificateArn": "arn:aws:acm:us-east-1:111122223333:certificate/3f1f4661-f01b-4eef-ae0d-123412341234",
            "SSLSupportMethod": "sni-only",
            "MinimumProtocolVersion": "TLSv1",
            "Certificate": "arn:aws:acm:us-east-1:111122223333:certificate/3f1f4661-f01b-4eef-ae0d-123412341234",
            "CertificateSource": "acm"

and then ran the command aws cloudfront create-distribution --cli-input-json file://cf.json.

Testing it

The first thing to test was simply curlling the newly created distribution – after waiting ages for it to actually be created. (I assume CloudFront is just continually working through a queue of distribution changes to each of the 40+ POPs, so it’s actually quite awesome.)

[[email protected] ~]$ curl -v
* Rebuilt URL to:
< HTTP/1.1 301 Moved Permanently
< Location:
< Server: AmazonS3
< X-Cache: Miss from cloudfront
* Connection #0 to host left intact
[[email protected] ~]$ curl -v
* Rebuilt URL to:
< HTTP/1.1 301 Moved Permanently
< Location:
< Server: AmazonS3
< Age: 11
< X-Cache: Hit from cloudfront

Would you look at that, both the HTTP & HTTPS connections worked!

So I changed the CNAME in CloudFlare to point to the new distribution, and tried again, this time with

[[email protected] ~]$ curl -v
* Rebuilt URL to:
< HTTP/1.1 301 Moved Permanently
< Location:
< Age: 577
< X-Cache: Hit from cloudfront
< Via: 1.1 (CloudFront)
< X-Amz-Cf-Id: kl8eZMDzH2BB7T3owENtjFkS2xtfcwqoOsZ4-SNxLY8LMdupbXrp9Q==
< X-Content-Type-Options: nosniff
< Server: cloudflare-nginx
< CF-RAY: 28d9413b3943302a-YYZ

Because I use CloudFlare’s caching layer, parts of the response are overwritten by CloudFlare (notably the Server section, among others). But we can still see that the request ultimately hit the CloudFront frontend. The request was still redirected, so everything looks good.

In Closing

The main thing I didn’t like about this setup? The obtuse documentation. I had far better grasp of everything working through the console, then applying that to the CLI. But there’s still tiny things.

  1. I hit the ancient ‘Conflicting Conditional Operation’ issue about S3 buckets being quick to be deleted from the console, but not actually deleted in the backend. So when I mistakenly thought that --region us-west-2 would be enough to get the S3 bucket to be created in us-west-2, it took ~1 hour to become useable again.

  2. Passing --region us-west-2 to aws s3api create-bucket will create a S3 bucket in the default region, us-east-1. You have to specifically pass --create-bucket-configuration LocationConstraint=us-west-2 to get it created in a specific region.

  3. There’s aws s3 and aws s3api. Why aren’t the s3api operations merged into s3? No other service has an api suffix.

  4. Having to trial and error to find out what parameters are required and what aren’t for different operations. The CloudFront create-distribution command was the worst offender of the commands I ran, just with the sheer number of parameters. I’m hoping the documentation improves before it comes out of beta.

  5. Weird UI bugs/features. Main one I noticed was the ACM certificate selector not being selectable unless a cert exists… and the refresh button is included in that. So the very first time I created a CloudFront distribution, I needed to refresh the page to be able to select the cert, losing my settings.

Now for the important question: Now that I’ve set it up, will I keep on using it? The simple answer is that I’m not sure. It’s nice that it’s offloaded to another provider that keeps it going without my intervention, and that after setting it up it won’t change. But at the same time, it’s another monthly expense. I’ve already put money into a server, and the extra load of a redirection config is negligible thanks to Ansible. There is some overhead with creating & managing Let’s Encrypt certs, particularly with the rate-limiting, and these redirects are entire subdomains, but I just set up an Ansible Let’s Encrypt playbook to run weekly, and the certs will be kept up to date.

It’s going to come down to how much I get billed for this.

No Comments

Using the Ansible Slurp module

I recently discovered the slurp module within Ansible when I was attempting to find new modules in Ansible 2.0. It is particularly interesting for me since I’ve been doing a bunch of stuff involving the contents of files on remote nodes for my OpenVPN playbook. So I decided to try using it in one of my latest playbooks and see how much better it is than doing command: cat <file>.

Using it

My usecase for slurp was checking if a newly bootstrapped host was Fedora 22, and upgrading it to Fedora 23 if it was. The problem in this case was that recent versions of Fedora don’t come with Python 2, so we can’t use gather facts to find the version of Fedora (and need to install Python2 before we do anything).

The suggested method was to install python using the raw command, and then run the setup module to make the facts available.

But I was going to reboot the node right after the install in any case, so I didn’t feel like running the full setup module, so this was a perfect place to try the slurp module.

Using it is simple – there’s only one parameter: src, the file you want to get the contents of.

Similarly, using the results is also simple, with one exception: The content of the file is base64 encoded, so it must be decoded before use. Thankfully, Ansible/Jinja2 provides the b64decode filter to easily get the contents into a usable form.

My final playbook ended up looking something like this:

gather_facts: no
    - name: install packages for ansible support
      raw: dnf -y -e0 -d0 install python python-dnf
    - name: Check for Fedora 22
        src: /etc/fedora-release
      register: fedora
    - name: Upgrade to Fedora 23
      command: dnf -y -e0 -d0 --releasever 23 distro-sync
      when: '"Fedora release 22" in fedora.content|b64decode'

Functionally, it’s pretty much identical to using the old command: cat <file> , register, and when: xyz in cmd.stdout style to get & use the contents of files. All of those elements are still there, just renamed at most – register is still being used unmodified.

The fact that I’m using a dedicated module for it though makes my playbook look a lot more Ansible-ish, which is something I like. (And the fact I don’t need to have a changed_when entry is a strong plus for code cleanliness.)

No Comments

Backing up & restoring Jenkins

I’m moving my jenkins instance to a new server, which means meaning up & restoring it.


The nice thing about it is that it’s almost entirely self-contained in /var/lib/jenkins, which means I really only have 1 directory to backup.

I’m using duply to back the folder up – but it’s 1.9GB in size. So to save space & bandwidth, I’m going to exclude certain files. This is the content of my /etc/duply/jenkins/exclude file:


The main thing I’m excluding is the build artifacts – because I’m building RPMs, the SRPMS are rather large (nginx-pagespeed SRPMs weigh in at 110+MB), so I exclude all files ending in .rpm.

Next, I’m excluding most of the stuff in the plugin folder. My reasoning behind this is that the plugins themselves are downloadable. However, Jenkins disables plugins/pins plugins to the currently installed version by creating empty files of the form <plugin>.jpi.disabled/<plugin>.jpi.pinned. I want these settings to carry over between versions. Unfortunately trying + **/plugins/*.jpi.pinned showed that everything else got removed from the backup. I’m assuming this is due to the use of an inclusive rule, so the default include got changed to default exclude.

In any case, I end up explicitly excluding things I don’t care about, which is probably good if something that I might need ever ends up in the plugins folder.

I also exclude workspace because everything can be recreated by building from specific git commits if need be. The job information is logged in jobs/, so I can easily find past commits even though the workspace itself no longer exists.

Finally, I also exclude the jenkins war folder. I believe that this is an unpacked version of the .war file that gets installed to /usr/lib/jenkins. It seems to get created when I start jenkins itself.

With just these 6 excludes, I’ve dropped the backup archive size down to <5 MB, which is a big win.

Normally I’d just take a live backup while Jenkins is running, but in this case where I’m moving servers, I completely shutdown Jenkins first, before taking a final backup with duply jenkins full.


For the restore, I first installed Jenkins using a Jenkins playbook from Ansible Galaxy. It’s fairly barebones, but it works well – and I don’t need to spend time developing my own playbook. I also installed duply, and I manually installed an updated version of duplicity for CentOS 7 from koji to get the latest fixes.

Once I got duply set back up, I restored all the files to a new folder with duply jenkins restore /root/jenkins. I restored it to a separate folder because duply appears to remove the entire destination folder if it exists, and I wanted to merge the two folders.

After the restore was complete, I ran rsync -rtl --remove-source-files /tmp/jenkins/ /var/lib/jenkins to merge the restored data into the newly installed Jenkins instance.

At this point, everything should have worked, except I was unable to login. After spending some time fruitlessly searching Google, I ran chown -R jenkins:jenkins /var/lib/jenkins, as the rsync didn’t preserve the file owner when it created the new files. Luckily enough, that fixed the problem, and I could now login.

I then spent a few hours working all this into an Ansible playbook so future moves are much easier.

, ,

No Comments