Conversation

Notices

1. Embed this notice
   Pete Corey (petecorey@mastodon.social)'s status on Thursday, 05-Oct-2023 18:25:52 JST Pete Corey

   Using Ecto schemas directly in migrations is a huge mistake. I hate my past self for doing it. Every time you modify a schema, all of your past migrations break.

   I've since switched to using schema-less queries in migrations, but I'm starting to be of the mind that migrations should just use raw SQL fragments...

   #elixir

   • Embed this notice
     clacke (clacke@libranet.de)'s status on Thursday, 05-Oct-2023 18:25:48 JST clacke

     in reply to

     @petecorey A huge mistake, but also a common mistake that probably everyone does once, whether they use Ecto, Alembic or Rails. 😊

     Depending on what the language and framework allows, an alternative to not using the ORM at all is to define the required objects locally in the migration.

     Rather than resorting to raw SQL I'm still quite in favor of using some abstractions in the application language that can be checked and composed with the usual tools, like the schemaless queries you mentioned.

     One big question is *which* SQL. With a little bit of SQLAlchemy in my Alembic migrations I can stay compatible with both SQLite and PostgreSQL, even when using advanced datatypes.

     12factors is nice and all, but it's also nice to have an in-memory db for certain integration tests.

   • Embed this notice
     Pete Corey (petecorey@mastodon.social)'s status on Thursday, 05-Oct-2023 22:04:38 JST Pete Corey

     in reply to

     • clacke

     @clacke That's a good point. Dropping down the raw SQL does lock me in to a specific tech unnecessarily. Sticking with schemaless Ecto queries is probably the way to go, I guess!

     clacke likes this.
   • Embed this notice
     Pete Corey (petecorey@mastodon.social)'s status on Monday, 09-Oct-2023 03:39:26 JST Pete Corey

     in reply to

     • Paweł Świątkowski

     @katafrakt Yeah, that’s the root of the problem here. It’s just so convenient! I imagine separating schema and data migrations and relying more heavily on feature switches is probably the most robust way to handle these things.

   • Embed this notice
     Jason Axelson :elixir: (axelson@fosstodon.org)'s status on Monday, 09-Oct-2023 03:39:26 JST Jason Axelson :elixir:

     in reply to

     @petecorey sometimes I’ll copy a version of the schema as it is at that time directly into the migration module. That way it’s static and won’t change.

     clacke likes this.
   • Embed this notice
     Paweł Świątkowski (katafrakt@genserver.social)'s status on Monday, 09-Oct-2023 03:39:32 JST Paweł Świątkowski

     in reply to
     @petecorey my go-to is to use migrations only for schema migrations and have separate machinery for data migrations. More on this here (although in Ruby context, but it applies to Ecto and others too): https://dev.to/katafrakt/data-migrations-in-regular-migrations-and-why-you-probably-should-not-do-that-2eem
   • Embed this notice
     clacke (clacke@libranet.de)'s status on Monday, 09-Oct-2023 03:45:05 JST clacke

     in reply to

     • Paweł Świątkowski

     @katafrakt @petecorey Fair points. Has someone formulated principles for how a schema migration should look for data migration to be separate?

     Some schema migrations are just schema migrations and some data migrations are just data migrations, but often you're migrating the data so that it fits in the new schema.

     I suppose it's like decoupling your frontend from your API or your API from your database schema, you'd have to have a concept of versioning, backwards compatibility, deprecation and breaking changes to your schema. So far I have gotten away with treating the API and schema as tightly coupled, the migration making sure they're always in lockstep.

   • Embed this notice
     clacke (clacke@libranet.de)'s status on Monday, 09-Oct-2023 03:46:32 JST clacke

     in reply to
     I don't know how an ORM would look that would handle this well. But then some people say we shouldn't be using ORMs and maybe this would be one of the reasons.
   • Embed this notice
     Daniel Barlow (dan@brvt.telent.net)'s status on Monday, 09-Oct-2023 10:21:28 JST Daniel Barlow

     in reply to

     • clacke
     • Paweł Świątkowski
     @clacke @katafrakt @petecorey "orm vs raw SQL" is a false dichotomy, really. You want some kind of abstraction over the relational model but you don't want the object mapping. I found this from first principles in CL and ended up creating "septeql" (old, unmaintained, zero users) but whether your language has something actually useful in this space is worth investigating. I don't know elixir but Sequel is nice for this in Ruby
     clacke likes this.
   • Embed this notice
     clacke (clacke@libranet.de)'s status on Monday, 09-Oct-2023 10:29:08 JST clacke

     in reply to

     • Daniel Barlow
     • Paweł Świątkowski

     @dan Turns out I do want the mapping though, warts and all. =)

     SQLAlchemy on Python has enough knobs and dials to overcome the worst of the impedance mismatch, "in particular the 'n loads' problem", and it has a nice clean layer between ORM and SQL abstraction for when you really need to get in there.

     The abstraction leaks but is mostly helpful.

     @petecorey @katafrakt

   • Embed this notice
     Paweł Świątkowski (katafrakt@genserver.social)'s status on Tuesday, 10-Oct-2023 10:18:36 JST Paweł Świątkowski

     in reply to

     • clacke
     @clacke @petecorey I don't have any principles formulated, but I usually just model data migrations as tasks (rake, mix, custom built triggered from admin panel...).
     
     Even with regular migrations during the deploy you have a timeframe when the migration is already run, but you're still serving old code (with large codebases it's non-negligible amount of time). So I have a rule that migrations need to be compatible with old and new code anyway.
     
     So you
     1. Deploy schema migration
     2. Run data migration
     3. Deploy new code that relies on migrated data
     
     Of course I can imagine situations where this does not cut it, but it works most of the time for me.
     clacke likes this.