Dobesland: Database schema migration with LiquiBase | Dobesland
Dmitri Livotov
· 1 year ago
Interesting tool. And in my projects I came to creating the DB-independent backup/restore tool, that backups the entire object model to a backup file and then restores it back, allowing to make a backup, update the software with dropping and re-creating a schema and finally restore data. This also brings another "bonus" - ability to move data across various database vendors easily.
dobes
· 1 year ago
Well, that seems like a nice approach, but it does mean that you have to perform migrations on your data dump/load schema. However, if the file format is written by hand, that's not too hard ... maybe not as hard as learning and using tools like these.
Definitely something to think about.
Dmitri Livotov
· 1 year ago
Yes, but we've used XStream (initially) to serialize POJOs from model to a our own backup file (coupled with trueZip this brings a solid single backup file transparently). In case, if in a new version of software, the object model changes, for instance, bringing a couple of new fields, this is handled transparently with XStream on de-serialize. Of course, in a complex model changes this requires to be supported in restoration code too, however, this is not very hard to maintain, especially, if model was accurately designed initially (it then does not require a lot of complex changes)
dobes
· 1 year ago
Thanks for bringing XStream to my attention, I hadn't seen that before. It looks very, very useful to me right now.
Ah sweet, they even have annotations! Awesome!
Marlin
· 1 year ago
We also chose to use hibernate with annotations so, as a small team of Java developers, we could focus on getting a great product out rather than on serializing and deserializing objects. That mostly worked, though Hibernate+Spring+Tapestry had some quirks (still does). As time goes on, we add new features, we refactor, and we have existing customer data to migrate to a new release...
To use a traditional ETL, we'd have to know more about how Hibernate maps things than we want to (and we already know too much). I didn't find anything that would help map one object graph to another, so I created my own Java program to read objects from the old database and write them to the new database.
To do so, the program has to have both the new and the old object definitions around, and hibernate sessions to the old and new databases. To reference the old objects, I renamed the top level package in the old version (com.foo to com.fooV230) and created a library with those classes. The migration (eclipse) project has this library and references the current code base with the new object schema, and the program that reads old objects, copies to new objects, does a little processing, and writes the new ones out to the new database. It's useful and it's painful. It's useful when there is some processing and default-setting to do from one version to another. It's painful to set up, ensuring that each and every field is copied correctly.
But once it is set up -- preferably before any model changes have been made to the target project -- the migration project is kept open by each developer and each developer must make changes to it when he or she changes the object model. If we've made hibernate annotation changes for performance or other reasons, the target database receives them transparently. At the end of development, we do some extra testing and clean up, and pass it off to the test team.
So far, it's worked, but I'm keeping my eyes open for a better way, or a tool that makes it simpler (requiring less high-level engineering time). I'm interested to hear how you solved your hibernate/annotations data migration...
dobes
· 1 year ago
Hi Marlin,
I've been using liquibase for my schema migrations with some success now.
My schema migration process goes like this:
1. Make changes to the hibernate annotations/schema
2. Test those changes using ejb3unit against a unit-testing-only database I call "ejb3unit", which is created by hibernate's schema export feature in drop-create mode; I configure ejb3unit to use that test database via its properties file.
3. Generate a liquibase diff from the last deployed database schema and the "ejb3unit" database.
4. Manually inspect the diff and decide whether anything was done that requires data conversion, and if so, insert custom classes and/or SQL that does the conversion and append that to the liquibase changelog I am using.
5. Deploy the new app to glassfish on my development machine, which will update the schema based on the new entries in the changelog, and test it manually to be sure the schema is correctly upgraded.
I feel pretty good about this methodology, I think it is robust because it incorporates unit testing into the process, and it doesn't require a lot of save/load work to migrate schemas.
I does require the occasional bit of SQL hacking for a migration, since I can't use hibernate in the data conversion classes, but most operations are just adding/removing fields.
All Comments
Definitely something to think about.
Ah sweet, they even have annotations! Awesome!
To use a traditional ETL, we'd have to know more about how Hibernate maps things than we want to (and we already know too much). I didn't find anything that would help map one object graph to another, so I created my own Java program to read objects from the old database and write them to the new database.
To do so, the program has to have both the new and the old object definitions around, and hibernate sessions to the old and new databases. To reference the old objects, I renamed the top level package in the old version (com.foo to com.fooV230) and created a library with those classes. The migration (eclipse) project has this library and references the current code base with the new object schema, and the program that reads old objects, copies to new objects, does a little processing, and writes the new ones out to the new database. It's useful and it's painful. It's useful when there is some processing and default-setting to do from one version to another. It's painful to set up, ensuring that each and every field is copied correctly.
But once it is set up -- preferably before any model changes have been made to the target project -- the migration project is kept open by each developer and each developer must make changes to it when he or she changes the object model. If we've made hibernate annotation changes for performance or other reasons, the target database receives them transparently. At the end of development, we do some extra testing and clean up, and pass it off to the test team.
So far, it's worked, but I'm keeping my eyes open for a better way, or a tool that makes it simpler (requiring less high-level engineering time). I'm interested to hear how you solved your hibernate/annotations data migration...
I've been using liquibase for my schema migrations with some success now.
My schema migration process goes like this:
1. Make changes to the hibernate annotations/schema
2. Test those changes using ejb3unit against a unit-testing-only database I call "ejb3unit", which is created by hibernate's schema export feature in drop-create mode; I configure ejb3unit to use that test database via its properties file.
3. Generate a liquibase diff from the last deployed database schema and the "ejb3unit" database.
4. Manually inspect the diff and decide whether anything was done that requires data conversion, and if so, insert custom classes and/or SQL that does the conversion and append that to the liquibase changelog I am using.
5. Deploy the new app to glassfish on my development machine, which will update the schema based on the new entries in the changelog, and test it manually to be sure the schema is correctly upgraded.
I feel pretty good about this methodology, I think it is robust because it incorporates unit testing into the process, and it doesn't require a lot of save/load work to migrate schemas.
I does require the occasional bit of SQL hacking for a migration, since I can't use hibernate in the data conversion classes, but most operations are just adding/removing fields.