|
|
| Line 242: |
Line 242: |
| | --> | | --> |
| | <!-- ************************* END OF OUTAGE INFO ************************* --> | | <!-- ************************* END OF OUTAGE INFO ************************* --> |
| − | <!-- TEMPLATES -->
| |
| − | <!-- TEMPLATE 1 START
| |
| | | | |
| | | | |
| − | <hr>
| |
| − | :'''WHAT:''' Linux kernel bug causing some of our Fedora 31 lab systems to hang.
| |
| − |
| |
| − | :'''WHEN:''' First identified in CS, last week of January, 2020. '''''UPDATES below the DETAILS section...'''''
| |
| − |
| |
| − | :'''DETAILS:'''
| |
| − |
| |
| − | ::As you know, recently, we upgraded our lab Linux clients to Fedora 31. Over the past several days, we've discovered some odd behavior from some of the lab systems. Upon further investigation, we have learned that there is a kernel bug that is causing some systems to hang (requiring a reboot) when certain NFS operations are performed. (NFS is the system that allows our Linux clients to access the home directories, certain apps, other shared file systems.)
| |
| − |
| |
| − | ::We're seeing that the bug is documented and that it affects not only Fedora 31, but some prior versions as well. It just happens to be that we are seeing its effects now.
| |
| − |
| |
| − | ::This is a Linux kernel issue and there are Linux folk working on trying to get this bug patcMarch 22hed, but we do not know how long it will take.
| |
| − |
| |
| − | ::In the meantime, CS IT is looking into work-arounds to bypass this bug as much as possible.
| |
| − |
| |
| − | ::There is no way to predict if or when this bug will cause a system to hang. Some of our systems may never notice it. We think it might happen to some systems that are heavily utilizing NFS operations, but again, we're not sure.
| |
| − |
| |
| − | ::Our recommendation is that when you're editing files, etc. save your work on a regular basis, just in case. This is a good practice to follow anyway.
| |
| − |
| |
| − | ::'''''IMPORTANT:''''' If you experience a system that hangs (and doesn't recover after several minutes), please email {{mailto|support}} to let us know the name of the system that experienced the hang, the time of the hang, and what activity you were doing on the system at that time (which will help us troubleshoot the problem further.)
| |
| − |
| |
| − | ::We will update here with our progress on circumventing this bug.
| |
| − |
| |
| − | ::'''UPDATE: 1/29/20''' We are applying alternative pre-bug kernels to those Linux clients that are available to be rebooted (to activate these kernels.) As more become available, we'll reboot them as well. We're hoping these alternative kernels will prevent the hangups until officially patched kernels become available.
| |
| − |
| |
| − | ::'''UPDATE: 2/7/20''' We have applied updates to most of our systems in order to activate the alternative pre-bug kernels, which we hope will prevent hangups. There are still some systems which have not yet had the alternative kernels applied, and that will happen as the systems become available for us to reboot.
| |
| − |
| |
| − | TEMPLATE 1 END-->
| |
| − | <!-- TEMPLATE 2 START
| |
| − |
| |
| − |
| |
| − | ===Suspended (Unavailable) ''Grad/Research Net'' services while rtfm.cs.jhu.edu & blaze.cs.jhu.edu are down:===
| |
| − |
| |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − |
| |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − |
| |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − |
| |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − |
| |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − |
| |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − |
| |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − |
| |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − |
| |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | TEMPLATE 2 ends -->
| |
| − | <!--:None at this time.-->
| |
| − | <!-- -->
| |
| − | <!--******************MORE TEMPLATES BEGIN HERE*****************-->
| |
| − | <!-- -->
| |
| − | <!-- ******TEMPLATE GRAD NET OUTAGE*****
| |
| − | ====Major CS Grad/Research Net service outage (mail, web, home directories, login, etc.) ====
| |
| − | <!----------------:None at this time.------------------------>
| |
| − | <!--;'''WHAT'''
| |
| − | :Major CS Grad/Research Net (and some Ugrad) IT services outage, as we update our file server system.
| |
| − | ;'''WHEN'''
| |
| − | :''Wednesday, June 13, from 5:30pm to approx 7:30pm.''
| |
| − | ;'''UPDATE'''
| |
| − | :June 13, 8:30pm. The service outage is complete, and are services our back to normal operation.
| |
| − | ;'''DETAILS'''
| |
| − | :Our CS Grad/Research Net home directory file server-related system, ''rtfm.cs.jhu.edu'', will be shut down so that important server-related updates can be applied. (In order to shut down that server, we must also shut down our mail, DNS, and other services server, ''blaze.cs.jhu.edu''.) We will apply the required updates on Wednesday, June 13, from 5:30pm to approximately 7:30pm.
| |
| − | :
| |
| − | :During the process of applying updates, several of our CS services will be unavailable. If there are any unexpected complications during the update process that will delay the restoration of services, we will update thi
| |
| − | {{outage
| |
| − | |time=9:00 am Monday, March 15
| |
| − | |services=ugradx
| |
| − | |summary=ugradx was temporarily unavailable for logins as we performed some software maintenance on it.
| |
| − | |update=As of 11:00 am, the maintenance is complete and ugradx is available again.
| |
| − | }}
| |
| − | s CS IT Support Wiki's Notices page.
| |
| − |
| |
| − | ===Suspended (Unavailable) ''Grad/Research Net'' services while rtfm.cs.jhu.edu & blaze.cs.jhu.edu are down:===
| |
| − |
| |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − |
| |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − |
| |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − |
| |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − |
| |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − |
| |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − |
| |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − |
| |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − |
| |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − |
| |
| − | : '''Note to Ugrad Net users:''' During the rtfm.cs.jhu.edu and blaze.cs.jhu.edu updates, email directed to your ugrad acct @cs.jhu.edu will not be immediately delivered. Other Ugrad Net services should remain available. -->
| |
| − | <!-- -->
| |
| − | <!--******************TEMPLATES BEGIN HERE*****************-->
| |
| − | <!-- -->
| |
| − | <!-- ******TEMPLATE GRAD NET OUTAGE*****
| |
| − | ====Major CS Grad/Research Net service outage (mail, web, home directories, login, etc.) ====
| |
| − |
| |
| − | '''WHAT:''' Major CS Grad/Research Net (and some Ugrad) IT services outage, as we apply security updates to our systems.
| |
| − |
| |
| − | '''WHEN:''' '''''Rescheduled for''''' Thursday, January 18, from 5am to approx 7am.
| |
| − |
| |
| − | '''DETAILS:''' Many CS Grad/Research Net servers will be shut down to apply security updates. We will apply these updates on Thursday, January 18, from 5am to approximately 7am.
| |
| − |
| |
| − | During the process of applying updates, several of our CS services will be unavailable. If there are any unexpected complications during the update process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at https://support.cs.jhu.edu/wiki/Notices
| |
| − |
| |
| − | Suspended (Unavailable) Grad/Research Net services while the servers are updated:
| |
| − |
| |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − |
| |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − |
| |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − |
| |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − |
| |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − |
| |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − |
| |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − |
| |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − |
| |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − |
| |
| − | Note to Ugrad Net users: During the Grad Net servers' security updates process, email directed to your ugrad acct @cs.jhu.edu will not be immediately delivered. Other Ugrad Net services should remain available. Also, we will be applying security patches on the CS Ugrad net the next day, so you'll see an email for that at some point soon.
| |
| − |
| |
| − | Updates, if necessary, will be posted here.
| |
| − | ******TEMPLATE END*****-->
| |
| − | <!-- -->
| |
| − | <!--******TEMPLATE BEGIN - UGRAD NET OUTAGE
| |
| − |
| |
| − | ::::'''''NOTE: DATE CHANGE BELOW for OUTAGE #1!!!!!!!!''''' -->
| |
| − | <!------------==== Major CS ''UGrad Net'' service outage (mail, web, home directories, login, etc.) ====
| |
| − |
| |
| − |
| |
| − | '''WHAT:''' Major CS Ugrad Net IT services outage, as we reboot our Ugrad home directory file server.
| |
| − |
| |
| − | '''WHEN:''' Friday, October 18 (Fall Break Day) from 9am to approx 10am.
| |
| − |
| |
| − | '''UPDATE''': As of 9:45 am, all affected systems are back in service.
| |
| − |
| |
| − | '''DETAILS:''' To patch a problem on our Ugrad home directory file server called barley, we need to reboot it. The boot process should take less than an hour.
| |
| − |
| |
| − | During the reboot process, several of our CS services will be unavailable. If there are any unexpected complications during the reboot process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at https://support.cs.jhu.edu/wiki/Notices
| |
| − |
| |
| − | Suspended (Unavailable) Ugrad Net services while barley is rebooted:
| |
| − |
| |
| − | * CS Ugrad Mail service, both incoming and outgoing, will be suspended for CS Ugrad Net users.
| |
| − |
| |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − |
| |
| − | * IMAP, POP, and Webmail from our CS (Ugrad Net) Mail server will be unavailable.
| |
| − |
| |
| − | * CS Ugrad Net webpages will be unavailable.
| |
| − |
| |
| − | * CS Ugrad Net user accounts/home directories will be inaccessible during the downtime.
| |
| − |
| |
| − | * ssh service to our CS Ugrad Net Linux boxes (ugrad1 through ugrad24, ugradx, and ugradz) will be unavailable.
| |
| − |
| |
| − | * Console (desktop) access to ugrad1 through ugrad24 will be unavailable.
| |
| − |
| |
| − | * MySQL service via the dbase.cs.jhu.edu server (Dr. Yarowsky's class) will be unavailable.
| |
| − |
| |
| − | Updates, if necessary, will be posted here. ------------------------>
| |
| − | <!--:'''WHAT:''' Emergency reboot of ''blaze'', the dept's main Grad/Research net mail server.
| |
| − |
| |
| − | :'''WHEN:''' ''(TODAY!)'' Tuesday, Dec. 12 @ noon and lasting approx 1/2 hour if all goes well.
| |
| − |
| |
| − | :'''DETAILS:'''
| |
| − |
| |
| − | ::We have discovered an issue with our mail spool disk on ''blaze'' our Grad/Research net mail server which must be corrected right away. As such, we expect that a configuration change and reboot should fix the problem.
| |
| − |
| |
| − | ::We plan to do this shortly, at ''noon today (lunchtime.)'' The reboot should take approx. 1/2 hour if all goes well. If there are issues, it might take some additional time to complete the repair.
| |
| − |
| |
| − | ::During the reboot time, there will be ''no CS email service in or out of the CS Grad/Research network.'' You will not be able to send or receive mail (IMAP/POP/etc.) through blaze (smtp.cs.jhu.edu/pop.cs.jhu.edu) or via webmail during that time. Also, DNS might be slow or non-existent, depending upon your client's own configuration.
| |
| − |
| |
| − | ::You also might experience slow connection times to www.cs.jhu.edu.
| |
| − |
| |
| − | ::Grad Net Lab Linux systems (masters1 through masters16, gradx, and gradz) may experience a short delay as their NIS service connects to a different server.
| |
| − |
| |
| − | ::Any updates and notice of completion will be presented through this NOTICES page today.
| |
| − |
| |
| − | :'''''UPDATE:''''' 12/12/17 at 12:15pm
| |
| − |
| |
| − | ::Reboot successful! Blaze is back in operation and services have been returned to normal operation.
| |
| − | -->
| |
| − |
| |
| − | <!-- :'''WHAT:''' Brief CS network outage
| |
| − |
| |
| − | :'''WHEN:''' ''Rescheduled for'' Thursday, July 20, from 9am to 9:30am.
| |
| − |
| |
| − | :'''DETAILS:''' JHU Networking (in conjunction with JHU Network Security) will be moving the various CS Networks in Malone Hall from one firewall platform to another. This will ''now'' take place on Thursday, July 20 at 9am and should only last approximately 5 minutes or so. But, we're providing a 30 minute window in case JHU Networking needs to troubleshoot any issues.
| |
| − |
| |
| − | :During the network outage, CS systems might not be seen by each other, and network traffic between CS and outside-of-CS may be halted. Many CS services will be unavailable when the network is unavailable. Again, JHU Networking expects this to be a brief network outage, so you might only experience a short system hang.
| |
| − |
| |
| − | We will update this Notices page in case there are delays with restoring any of our services.
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | -->
| |
| | <!-- ****** FILE SERVER OUTAGE TEMPLATE ****** | | <!-- ****** FILE SERVER OUTAGE TEMPLATE ****** |
| | | | |
