Major Feature request: Make major upgrades smooth going

[hkais]

Before I am getting started, I am appreciating that Znuny is taking over OTRS code to bring it to a more productive environment.
And I really hope Znuny will clean up the messes from the old OTRS especially considering existing users.

Please consider this feedback as constructive feedback, but maybe a very direct pin pointing to IMHO the biggest issues I see in Znuny for the future success. So please consider this post as a lens to a bigger issue and also as my way to help to evolve Znuny for the future.

I am using now OTRS since over 16 years now. And to sum up:
I have NEVER had a smooth major version upgrade


I had already multiple tasks to analyze how drop OTRS/Znuny and to replace it with other software which keeps upgrades running smoothly.
My management coming from software engineering and saying that it is like building a house on top of a basement, but the basement is somehow built together without a clean maintenance and change-management plan.

What are the pains?
In the biggest environment I am supporting (about 10Mio Articles in the database) the Upgrade process takes ages. This is okay if the upgrade would run smoothly

But in fact the upgrade is failing regularly and mostly on very basic issues, which considered from informatics/software engineering looks like obvious things are left aside during programming.
IMHO the major goal shall be on programming on such an "old" system to keep its admins being able to maintain and being able to do upgrades on their own.

Let me sum up from memory
E.g. from 1 to 5, I had in each major migration the issue that the schema/DDL was not well maintained, and I had to fix some minor steps.
But - from side line - is it not the elementary thing to ensure that changes on the basement of the house the database will be always published to the source repository? Means to have a n+1 upgrade path for every change on the DDL?
I know from tools like e.g liquibase, which provide a sort of source repository atop of a database to keep the changes always up to date on the database.

The next thing are checking of preconditions. Sadly these preconditions are checked in the execution steps. Means some minor checks are done before starting upgrade, and some bigger tasks are getting executed and afterwards or implicitly some addtional checks are done.
Now consider, your upgrade takes ages (okay hours) since the database changes are taking a very long time.
Now your upgrade fails due to the lack of missing documentation? You hate it
You say, okay let me fix it and rerun the upgrade the next day.
But what happens again, the upgrade runs now maybe some steps further an fails again.
Now you hate it even more
You fix it once more, sometimes you do not have even a clue what the error is and you need support on an upgrade.

Now from professional perspective, do you want to mess around with any microsoft update/upgrade of an previous version to a later version? Surely not, would be great if Znuny would take this also as a major professional and software engineering task to ensure that the upgrade run smoothly.

E.g. it would already help a lot to ensure all precondition checks are done before the time and IO consuming opperations get started.
E.g. if you have renamed namespaces or Filenames from one version to another, and this depends on some settings to work well:
where is the issue to check these preconditions with a few lines of code before you start with the full migration?

Even better would be:
your developers have modified it, why are they not able to ensure that the preconditions are also migrated properly?

To understand this post better, please check all this upgrade issues from otrs 5.0.42 to latest znuny 6.0.38
viewtopic.php?f=62&t=42963
viewtopic.php?f=62&t=42967
viewtopic.php?f=62&t=42964
viewtopic.php?f=62&t=42965 (would help to reduce the time needed on failure)
viewtopic.php?f=62&t=42961
viewtopic.php?f=62&t=42957
viewtopic.php?f=62&t=42958

Now you could say, keep the upgrades nearer to the release dates. But here would be the question back:
If you have a highly fragile upgrade process, how much willing and how much joy do you have to upgrade to new major version, if you know it will take you muliple nights and multiple days to get the software gone through the upgrade process?

Hope my message is not considered as Znuny is crap, instead I hope that the tech lead will consider what a good software engineering delivery is and would also consider things like:
- collect upgrade issues from the community, e.g. by a reporting system which reports automatically errors to your bug tracker on upgrade
- use these reports to find a better migration solution
- consider to do more thourough upgrade checks before doing the time consuming tasks
- consider how to change your development process to get a clean upgrade process for each source change

[root]

Hi,

Let me answer wherever I can with the best of my knowledge and experience.

I have NEVER had a smooth major version upgrade

I had them, and not just a few. Truth is there were also bad ones I never forget. But the root causes was always the same.

IMHO the major goal shall be on programming on such an "old" system to keep its admins being able to maintain and being able to do upgrades on their own.

I fully agree to this. Often this is not possible because of undocumented changes. The primary goal is always to identify the first.

Let me sum up from memory
E.g. from 1 to 5, I had in each major migration the issue that the schema/DDL was not well maintained, and I had to fix some minor steps.
But - from side line - is it not the elementary thing to ensure that changes on the basement of the house the database will be always published to the source repository? Means to have a n+1 upgrade path for every change on the DDL?

There is only one version, 3.x.y, from where I know that the DDL was not honored during the migration. In some version DDL issues can be fixed by applying the latest available patch level when you start. E.g. You start with a 3.2.1 you go first 3.2.9. And from this point on it's always the latest PL of every step in between.

The next thing are checking of preconditions. Sadly these preconditions are checked in the execution steps. Means some minor checks are done before starting upgrade, and some bigger tasks are getting executed and afterwards or implicitly some addtional checks are done.

I fully agree on this, but not everything what you consider a precondition can be checked at the beginning. But were open have issues filed at GitHub

Even better would be:
your developers have modified it, why are they not able to ensure that the preconditions are also migrated properly?

This one needs an explanation for me.

To understand this post better, please check all this upgrade issues from otrs 5.0.42 to latest znuny 6.0.38
viewtopic.php?f=62&t=42963
viewtopic.php?f=62&t=42967
viewtopic.php?f=62&t=42964
viewtopic.php?f=62&t=42965 (would help to reduce the time needed on failure)
viewtopic.php?f=62&t=42961
viewtopic.php?f=62&t=42957
viewtopic.php?f=62&t=42958

I think I checked them all and have answered or will wherever I can contributed sth. But using utf8mb4 e.g. was never written anywhere. if you use the installer the charset is the right one and for the manually install the manual stated utf8. And as far as I remember there was a check for the proper character set in an earlier version. This looks to me that the database was updated without checking the pre-conditions.

Hope my message is not considered as Znuny is crap, instead I hope that the tech lead will consider what a good software engineering delivery is and would also consider things like:
- collect upgrade issues from the community, e.g. by a reporting system which reports automatically errors to your bug tracker on upgrade
- use these reports to find a better migration solution
- consider to do more thourough upgrade checks before doing the time consuming tasks
- consider how to change your development process to get a clean upgrade process for each source change

The migration is something were we always work on improvements. We always accept bug reports via https://github.com/znuny/znuny/issues, checking this forum on oru own and create bug reports if we see them.

But at the end of the day, my experience is that not many users share their experience from migrations. They only want to get over it. This is nothing what help us at all. This is why I say: Thank you for you time and effort to provide feedback.

- Roy

[cilap]

Hi there,

I am jumping in the discussion to bring in my 2ct.
Please do not consider it as blaming or ranting, it is to point to common issues on development.

Let me answer wherever I can with the best of my knowledge and experience.

I have NEVER had a smooth major version upgrade

I had them, and not just a few. Truth is there were also bad ones I never forget. But the root causes was always the same.

I fully appreciate that you are providing help for the community in such cases.
From business perspective, I cannot understand why the issues faced with migrations are not getting incorporated into a doc or better the source code.
E.g. a section on the website, or on the doc of Znuny named
Known Upgrade issues would be great
or even better
the issues are getting fixed in code and are getting provided to avoid repeating migration errors
From what I have read here all issues seem to be self made issues (but I am not well in the code level, only judging from the posts)

The later would improve the migration experience to every admin of Znuny and would also make the migration more mature. Beside of the side effect that the support requirements here in the forum go down to really bad issues.

IMHO the major goal shall be on programming on such an "old" system to keep its admins being able to maintain and being able to do upgrades on their own.

I fully agree to this. Often this is not possible because of undocumented changes. The primary goal is always to identify the first.

Maybe I do not understand something well here, but developers do changes, and if there are clean developer guidelines and definitions of done, the developers cannot have undocumented changes. If you refer to undocumented changes from the admin side, here the question would arise how an admin can do undocumented changes.

Let me sum up from memory
E.g. from 1 to 5, I had in each major migration the issue that the schema/DDL was not well maintained, and I had to fix some minor steps.
But - from side line - is it not the elementary thing to ensure that changes on the basement of the house the database will be always published to the source repository? Means to have a n+1 upgrade path for every change on the DDL?

There is only one version, 3.x.y, from where I know that the DDL was not honored during the migration. In some version DDL issues can be fixed by applying the latest available patch level when you start. E.g. You start with a 3.2.1 you go first 3.2.9. And from this point on it's always the latest PL of every step in between.

In that time I have done some migrations and I can remember of tons of retries with wrong DDLs. Even had cases there I had to interrupt migrations to be able to trick DDL changes so that later DDL-SQLs will work well. Was a really hairy migration. And left on my end a very bad emotion.
But as we all now see a better chance for the future of Znuny I am willing to support here too. So let us forget the past and look forward.
For me means looking forward to have a clean, mature and stable migrations for every outside environment as goal (of course there may be still bad cases) The goal shall be to have a neat and fluent migration, like we are all used from other professional software solutions.
And from my past experiences the root cause for not having this is:
- developers do not do mandatory steps, e.g.
- on file renaming, to ensure to have migration scripts
- on DDL changes, to ensure the developers incorporate these changes with the migration admins are getting delivered
- on config changes to ensure this changes are also done with migration scripts
- also having a systemized error reporting from migration issues, which maybe send a e-mail or even better are creating automated issues, would massively improve the quality of the migrations.

This is often a matter of having clean developer guidelines along with mandatory processes for developers and of course persons who do acccordingly the code reviews (QA)

The next thing are checking of preconditions. Sadly these preconditions are checked in the execution steps. Means some minor checks are done before starting upgrade, and some bigger tasks are getting executed and afterwards or implicitly some addtional checks are done.

I fully agree on this, but not everything what you consider a precondition can be checked at the beginning. But were open have issues filed at GitHub

fully agree, see also my previous part about error reporting system

To understand this post better, please check all this upgrade issues from otrs 5.0.42 to latest znuny 6.0.38
viewtopic.php?f=62&t=42963
viewtopic.php?f=62&t=42967
viewtopic.php?f=62&t=42964
viewtopic.php?f=62&t=42965 (would help to reduce the time needed on failure)
viewtopic.php?f=62&t=42961
viewtopic.php?f=62&t=42957
viewtopic.php?f=62&t=42958

I think I checked them all and have answered or will wherever I can contributed sth. But using utf8mb4 e.g. was never written anywhere. if you use the installer the charset is the right one and for the manually install the manual stated utf8. And as far as I remember there was a check for the proper character set in an earlier version. This looks to me that the database was updated without checking the pre-conditions.

Compared to the past the support is great here! Many thanks for the fresh attitude here with Znuny!

One thing pops up to me.The doc refers that OTRS requires UTF-8 encoding, but it does not name specifically which encoding per database.
E.g. if there would have been cleanly stated use utf8_general_ci or better to have the SQL for it, it would be clear and without a path of bad interpretation.
If you consider the point here:
okay I need to use UTF-8,
google for how to set to UTF-8,
you will find descriptions to use utf8mb4, since other is marked as buggy and deprecated, everyone would choose for utf8mb4
And here my suggestion would be:
developer know what shall be defined, since the implemented a check. Instead of
your database needs to be UTF-8 it would be better to name
your database has the encoding Xy, the right encoding on your database MySQL/MariaDB would be utf8_general_ci. You can set this with the command "the command"
It costs the developer who implemented the check a minute more of work to provide reasonable and understandable messages to admins. And it wouild also be a well documented migration, since it guides the admin thouroughly and clear through the process.

Hope my message is not considered as Znuny is crap, instead I hope that the tech lead will consider what a good software engineering delivery is and would also consider things like:
- collect upgrade issues from the community, e.g. by a reporting system which reports automatically errors to your bug tracker on upgrade
- use these reports to find a better migration solution
- consider to do more thourough upgrade checks before doing the time consuming tasks
- consider how to change your development process to get a clean upgrade process for each source change

The migration is something were we always work on improvements. We always accept bug reports via https://github.com/znuny/znuny/issues, checking this forum on our own and create bug reports if we see them.

But at the end of the day, my experience is that not many users share their experience from migrations. They only want to get over it. This is nothing what help us at all. This is why I say: Thank you for you time and effort to provide feedback.

Fully agree to improving for future.
Please consider to take well established and commonly available technologies, and well known software development principles, which are targetting especially the cases of keeping sources in sync with the requirements on config management and change management on DDLs.

best wishes

[root]

IMHO the major goal shall be on programming on such an "old" system to keep its admins being able to maintain and being able to do upgrades on their own.

I fully agree to this. Often this is not possible because of undocumented changes. The primary goal is always to identify the first.

Maybe I do not understand something well here, but developers do changes, and if there are clean developer guidelines and definitions of done, the developers cannot have undocumented changes. If you refer to undocumented changes from the admin side, here the question would arise how an admin can do undocumented changes.

Hi,

Here I really refer to undocumented changes in Perl code, templates and configuration changes with side effects. I saw and see this often in migration projects combined with an "oh, I forgot to mention". This why we ALWAYS recommend to package customized and added files (logos, cronjobs), this is a simple and easy task.
During development we have many guidelines to prevent typical pitfalls, e.g. this is one of the reason why we insist on using the code policy and unit tests. And every entry in the CHANGES.md is also checked if a change to the documentation is required.

- Roy

[Johannes]

Hi,

thanks for the time you put into your post here, but we as the team, feel that we have to chime in here.

Migrations:

Migration are a complex and very personal task which require a deep working knowledge of the system which you desire to update/migrate. From OTRS 2.0 to Znuny 6.2 there have been continual improvements to make this easier. One example is that we have a database migration script and no single SQL files anymore. This tool has been renamed to Migrate*.pl because it does much more to make life easier.

The following parts of the framework have massively changed and require constant reeducation:

- perl API
- database handling/structure
- template engine
- upgrade process itself

These are just a few examples.

There are mechanisms in place for handling customizations to the framework:

- Package Management (since 2.x)
- Project Provided
- Vendor Provided
- Self-built
- Local Custom Folder (since 3.0)

As we here (and many other users and support companies worldwide) have smooth migrations every day on systems of any size and complexity. We can safely say that it's a question of planing an experience. It's even automated on most systems!

Many service providers are out there who can help you with migrations or give you trainings. We strongly reccomend these to you in your situation. It seems that the knowledge, you gained in this field, is unfortunately not enough to cover the possible pitfalls of a migration. Or you have the wrong expecations. Especially when you customized your instance.

Additionally, you should:

- Package your customizations
- Regularly read the documentation (before a migration)
- Use or define good practice for customization in your special use case.
- Perform at least one test migration
- Start asking questions / raise issues beforehand on Discord or here in the forum

This is not a simple plug-and-play software, but a powerful tool!

Agreed:

- The migration must be orchestrated (Step by step in a certain order)
- Software has bugs! -> You report, we fix.

As we don't have a 5 digit list of issues/forum about migration issues and we handle, as maintainers, hundreds of systems and maintain the code over a long period of time.
We don't really see the huge issues you do.

Continual improvement is our goal. Some things take time. You see an issue, report it! (thanks for that) If it is valid, we fix it.

The community and our installation base are a big part of these improvements and innovation.

In this, some things are out of our control like:

Why MySQL / MariaDB call utf8 -> utf8 and not utf8_mb_3.

But the problem is related to this specific database product. Even now, where the new default is utf8_mb_4, which is proper utf8 with full 4 Byte, they call utf8mb3 utf8.

In light of this, we have the change for the new MySQL / MariaDB default on the list since day one. It's not a high priority due to a low volume of requests/demand for this change.

My takeaway is that we will add a special note to the documentation explaining this a bit more.

Your takeaway is we, and others, recommend PostgreSQL in production if you use MySQL / MariaDB you have deal with it's specifics.

We admit therefore, we might underestimate the issue for MySQL / MariaDB users who do not deal daily with their DBMS.

In the future:

We have major changes regarding the plugin and migration mechanisms on the list for the upcoming years. The new search, the new UI and other things are just more important at the moment, then the noted MySQL issue.

The upcoming changes to the plugin management are going to cause you headache as well, if you customize outside of the recommended ways.

Anything less, will lead to failure!

That is why we are relasing the updated developer documentation alongside the Znuny 6.3 release.

We also push all the documentation repositories to GitHub. This commitment by us makes it easier for to track possible issues and accept suggestions for changes to the documentation. Commenting and suggesting changes to the documentation is your commitment as a community member, and not distractive and disruptive complaints in the forum about how the software doesn't allow you to point and shoot your update.

If you are not happy with the pace, I'm sorry. Change takes time:

We must consider the wide-spread (hundreds of thousands) usage and cover all the bases to keep the softawre stable for our entire user base.

Regards
Shawn & Johannes
(In the name of the the Znuny Team)