Continuous Integration and Deployment - Who's doing what ?

[Mitch C. Amiano]

I'm currently working on applications in Wolftech I.T., and have been working on a lightweight release process.

The historical approach is to initiate a project in a beta folder with access to production filesystem and database resources. While I've migrated the apps I'm working on into a fully isolated development environment, there are still  gaps to bridge in deployment.

I'd like to hear how others are addressing software release processes:

- Version control, particularly release tagging/branching

- Continuous Integration, including automated testing (Is anyone using Jenkins for C.I. ?)

- Deployment automation, including database migrations and releasing patches

[Jonathan Champ]

Some things we've done:

• operate on a copy of the database and filesystem (zfs snapshots make this much easier)
• using SVN or git for new version deployment (update the branch/tag in a neutral location so that the server can do a simple update/rebase)

[Mitch C. Amiano]

ZFS sounds interesting.

I am thinking in a deployment context, that a fetch+hard reset on origin+tag checkout would be preferable to a pull/merge or rebase. Merge and rebase both have the side-effect of creating new commits and changed hashrefs, whereas reset updates the HEAD reference to point to an existing commit.

[Jonathan Champ]

Yes, fetch with hard reset is the right way to go. One of the options for pull makes this easy if you haven't made any local edits: git pull --rebase

I was referring to rebase --onto for the neutral copy change management so that you can keep a feature branch synced with master until the feature branch is completed.

[Mitch C. Amiano]

Got it on the branch syncing. I don't think a reset --hard is equivalent to a pull --rebase; the latter could have an unresolved conflict from history rewriting on the remote or a local commit. Conflicts are uncommon but still potentially there. 

[Jonathan Champ]

Right, I was lumping history differences into "edits". That's what I get for typing too fast, ha!

[Mitch C. Amiano]

Oh, and thanks for the reply.

[Jason Austin]

This is a big deal to us, so we are working on changing up our process.  Here is what we do now, and then where we are planning to go very soon.

What we do now...

• We all have local development environments on OS X.  We are using Zend Server CE, but MAMP would work as well.
• All our source code is under version control in SVN or Git.
• All our production apps run out of AFS
• We have a staging environment setup in AFS that we deploy to first.  We tag our code in SVN or Git, and then export out directly to the staging environment.
• We have staging databases setup as well.  We copy the production DB to the staging DB, then run any DB migrations we have created for the new release.
• We have a script that sets permissions on the correct directories.
• We test in stage
• Once we are satisfied with staging, we do the entire process again into production.

This has worked for us for a few years, but it has become a huge pain lately as we are doing a lot more deployments.  It (usually) takes 2 of us to do all of this stuff in a timely manner, and there is a lot of areas for screwing up.  So, I'm currently working on moving to something more automated.

Where we are going...

• We are working on VM's that exactly mirror our production environment that each dev can run on their local machines.  This will ensure that we are all on the same development environment  and that the environment we are developing for will be exactly that as production.
• All of our source code will remain in Git or SVN, but we are going to start migrating all of our applications to GitHub private repos as they are way easier to manage than SVN, especially when you are dealing with naive developers like part time students.  The process of pull requests is much more simple than dealing with patch files like we do now.
• We have created some phing scripts to handle external dependencies, file permissions, and the execution of our DB migration script.  The process would be:  Checkout the code -> run the phing script -> profit.
• We are going to change up how we do deployments to AFS to take advantage of symlinks.  What we will do is deploy a tag to a releases directory (/releases/application-name/1.0.0).  Then, in our staging or production environments, use symlinks to map the directory to the release folder.  What that means is that once we get everything the way we want it in staging, we just have to change the symlink over for production and the same code will execute.  This limits downtime and makes it stupid simple to roll back.
• I am currently working on getting a jenkins environment going so that all of this can be automated.  

As far as testing, we have unit tests written for a good chunk of our core framework, but we need to do better about it for the rest of our code.  All our unit tests would be integrated as part of our jenkins environment.

We are in the infancy of our migration to our new way of doing things, but I will report back once we have things setup.

[Mitch C. Amiano]

Have you looked at Vagrant?

How does one get symlinks to be visible to the Web server on AFS? Or do you have some control over the lockdown on link following (afaict it is no follow on what I've seen).

[Jason Austin]

Garrison was the one working on the environment deployment, and I think he was using Puppet, so I don't really know about Vagrant.

[Mitch C. Amiano]

Vagrant is more about making isolated, reproducible development environments using VirtualBox. Puppet/chef or even shell scripts can be used to provision the environments. We were thinking it could be a good tool for setting up environments for part-timers and work studies, and as a workaround for those who otherwise prefer the  DOS Windows platform. 

[Jason Austin]

As far as symlinks, we're likely working on very different versions of webservers if you are on Engineering's webservers.  They are a much different animal than the rest of the webservers from OIT.

[Mitch C. Amiano]

Using symlinks for deployment staging and cutover was one of the first things I looked at (and it certainly doesn't work on our servers). I didn't want to believe it, but I do understand the motivation for the prohibition.