Greetings everyone.
I would like to provide a status update on the datacenter move.
On the plus side, we have all our data migrated to the new datacenter.
All the databases and storage are available in the new location.
Also, many of our services are back and functional.
However, the build system pipeline is not yet ready to come back up.
We are trying to debug/fix an authentication issue that is preventing
ssh git access and lookaside uploads. As soon as that issue is solved,
we can continue the build system bringup. That will consist of:
* Testing official build for signing, bodhi handling and gating testing
* opening koji and pkgs/src ssh.
* Testing updates composes
* Testing rawhide composes
I know packagers are eager to resume their work, hopefully
we will have things back up soon.
Issues we know about and are working on:
* The above dist git auth issues.
* mailman is sometimes hitting out of memory and needs to be restarted.
* Our meeting bot on matrix is unable to write log files to it's storage.
* a few smaller applications haven't yet been redeployed or need some fixes.
Unrelated, as we were moving, the AI scrapers started really hitting
pagure.io (which was not moving or changing due to the move) which made
things pretty difficult. We have blocked the 'blame' and 'history' web
enpoints. If you need those, you should be able to clone the git repo
and do them locally. This has moved the load back down to manageable.
If you are trying to use the new src.fedoraproject.org server, you may have
noticed it's ssh host keys changed. In the past we kept the old keys
but this time we made a clean break. You can verify the keys by getting:
https://admin.fedoraproject.org/ssh_known_hosts and adding it to
your local .ssh/known_hosts. This will allow ssh to see the new host
keys are signed with our CA. You can also use our sshfp records
if you are using a dnssec enabled resolver.
If you have noticed some other problem not mentioned above:
If the issue is minor, consider just waiting and letting us know/filing
a ticket on it next week if it's still happening.
If it's something major, do let us know:
https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/
and we will try and fix it as soon as we are able.
Keep in mind we are focused on the above issues and will need
to prioritze our time.
We will also be having a retrospective and sharing what we learned after
everything is back in a good state.
Thanks again for your patience.
kevin
--
_______________________________________________
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
No comments:
Post a Comment