|
|
| (699 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| | <!-- *************** BEGIN ACTIVE NOTICES **************************** --> | | <!-- *************** BEGIN ACTIVE NOTICES **************************** --> |
| | | | |
| − | == <span style="color:#ff0000">CS IT STAFF AVAILABILITY</span> (New!) == | + | == {{red|CS IT STAFF AVAILABILITY}} == |
| − | <br>
| |
| | | | |
| − | :<span style="color:#ff0000">NOTE:</span> Beginning March 18, 2020, and until further notice, the CS IT Support Staff will '''not''' be available ''in-person''. We will continue being available ''M-F, 8:30 to 5pm'', but '''online''' only. Therefore, please remember to email [mailto:support@cs.jhu.edu support@cs.jhu.edu] for assistance. Also, please have [https://support.cs.jhu.edu/wiki/Zoom Zoom] available on your device(s), so that we can arrange to contact you using that method, as an alternative to in-person communication. We are also looking into providing a virtual version of Steve DeBlasio's daily office hours for accounts, etc.
| + | * The CS IT Support staff is back in Malone Hall. Our group is working a hybrid schedule with some of us working remotely on certain days, while others of our team are on campus. |
| | + | * We continue being available M-F, 8:30 to 5pm, but there may be certain times we might be online only. |
| | + | * Therefore, please remember to email [mailto:support@cs.jhu.edu support@cs.jhu.edu] for assistance. |
| | + | * Also, please have [https://support.cs.jhu.edu/wiki/Zoom Zoom] available on your device(s), so that we can arrange to contact you using that method, as an alternative to in-person communication. |
| | + | * For the time being, Steve DeBlasio, our accounts liaison, is providing a virtual version of his daily office hours for accounts, etc. Email [mailto:support@cs.jhu.edu support@cs.jhu.edu] for details. |
| | | | |
| | <br><p> | | <br><p> |
| | | | |
| − | == <span style="color:#ff0000">SERVICE OUTAGES <span>==
| |
| | | | |
| | + | <!-- |
| | + | == {{red|JHU IT SERVICE OUTAGES AFFECTING CS IT SERVICES}} == |
| | + | === PLANNED ''JHU'' SERVICE OUTAGES === |
| | + | --> |
| | <!-- | | <!-- |
| | === UNPLANNED ''JHU'' SERVICE OUTAGES === | | === UNPLANNED ''JHU'' SERVICE OUTAGES === |
| − |
| |
| − | ::'''WHAT:''' ZOOM login issues.</span>
| |
| − |
| |
| − | ::'''WHEN:''' Dec 14, 2020
| |
| − |
| |
| − | ::'''DETAILS:''' Zoom has been problematic for many WSE/CS users as of this morning. If you are having issues logging in, please contact [https://support.cs.jhu.edu/wiki/Contacting_WSE_IT_Support WSE IT Support]
| |
| − |
| |
| − | ::ALso, make sure you're logging into Zoom correctly, if you had been logged out. Please visit our [https://support.cs.jhu.edu/wiki/Zoom_For_Video_Conferencing Zoom for Video Conferencing] page for details.
| |
| − |
| |
| − | <br>
| |
| | --> | | --> |
| | | | |
| − | === UNPLANNED CS SERVICE OUTAGES === | + | == {{red|CS IT SERVICE OUTAGES (Unplanned & Planned)}} == |
| − | | |
| − | <!--====<span style="color:#0000ff">PHISHING EMAIL AND OTHER ACCOUNT ISSUES!</span>====
| |
| − | | |
| − | <div style="color:#0000ff">
| |
| − | :5/5/20: We are currently receiving a large number of phishing emails directed at the CS Department. If you receive an email with links in it and have ''any'' doubts about the email, '''do not click the link'''. You '''do not''' need to send the emails to us at this time, as we are actively tracking the situation.
| |
| − | | |
| − | ::Also...
| |
| − | | |
| − | :Because of these phishing attacks, if you've followed the phishing links and provided your account credentials to others, and then find you are unable to login to your CS account get your CS email, then, please see our [https://support.cs.jhu.edu/wiki/How_To_Reset_Your_Password password reset procedures]. If your CS email account has been disabled, and you need you need to contact us, you will need an alternate non-CS email address to do so. Also, if you did follow a phishing link and enter your account credentials and then realize what happened, please email [mailto:support@cs.jhu.edu support@cs.jhu.edu] '''''right away''''', and let us know.
| |
| − | | |
| − | </div>-->
| |
| − | | |
| − | <!-- :None at this time. -->
| |
| − | | |
| − | :'''SELF-SERVICE FILE RECOVERY ISSUES'''
| |
| − | | |
| − | ::30 October 2020:
| |
| − | | |
| − | ::The <code>/users/backup</code> directory used for [[Self-Service File Recovery]] on the CS '''''Grad''''' Net Linux clients may be sporadically unavailable, due to a software bug. We're working on getting a fix for it. But, it might be some time before a fix is made available. Therefore, until further notice, if you need a file restored and cannot access the <code>/users/backups</code> directory, please [[BACKUPS - Requesting Restoration of a Deleted File|request restoration from our offline backups]].
| |
| | | | |
| − | ::Self-Service File Recovery continues to be available on the CS '''''Ugrad''''' Net.
| |
| | | | |
| − | <br>
| + | === UNPLANNED CS IT SERVICE OUTAGES === |
| | | | |
| − | === PLANNED CS SERVICE OUTAGES ===
| |
| − |
| |
| − | <br>
| |
| | :None at this time. | | :None at this time. |
| − | <!--
| |
| | | | |
| − | ==== Major CS Grad/Research Net service outage (mail, web, home directories, login, etc.) ==== | + | === PLANNED CS IT SERVICE OUTAGES === |
| | | | |
| − | '''WHAT:''' Major CS Grad/Research Net IT services outage, as we update our file server system.
| + | <!--:None at this time.--> |
| − | | |
| − | '''WHEN:''' ''Tuesday morning, Oct 13, 5am to approx. 6am.''
| |
| − | | |
| − | '''DETAILS:'''
| |
| − | :Our CS Grad/Research Net home directory file server, ''rtfm.cs.jhu.edu'', will be rebooted so that a time-sensitive, required update can be applied. We will apply the required update this coming Tuesday morning, Oct 13, starting at 5am. We expect the reboot and configuration to take no more than an hour if all goes well.
| |
| | | | |
| − | : During the process of applying the update, several of our CS services will be unavailable. A list is below.
| + | ==== Malone '''MH 122''' ''and'' '''G61''' Ugrad and Masters Labs {{red|'''''Closure'''''}}==== |
| | | | |
| − | : If there are any unexpected complications during the update process that will delay the restoration of services, we will update this CS IT Support Wiki's Notices page. | + | :'''WHAT:''' Malone Hall MH 122 '''and''' Malone Hall G61, both the Masters and Ugrad sides of the lab, '''and ''all'' the lab Linux computers, will be closed to CS students due to an ACM Programming Contest. |
| | | | |
| − | ===Suspended (Unavailable) ''Grad/Research Net'' services expected to be down while rtfm.cs.jhu.edu is rebooted:===
| + | :'''WHEN:''' Starting Friday evening, November 15, 5pm '''through''' Saturday, November 16, 9pm. |
| − | | |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users. Incoming mail should be queued by the sender until our mail service is back up again.
| |
| − | | |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − | | |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − | | |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − | | |
| − | * SMB File Shares (CS Office Staff and some CS faculty and grad students use these) will be unavailable.
| |
| − | | |
| − | * SSH service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − | | |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable. However, since users are not currently allowed in the CS labs, this should not be an issue.
| |
| − | | |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − | | |
| − | :'''Note:''' CS ''Ugrad Net'' services should not be affected by this outage other than incoming Ugrad Net email might be delayed.
| |
| − | | |
| − | -->
| |
| − | <!--
| |
| − | | |
| − | :'''WHAT:''' CS Ugrad systems will be unavailable during a server upgrade.
| |
| − | | |
| − | :'''WHEN:''' Monday, June 29, starting at 8:30am and lasting approx. 10:30am. | |
| − | | |
| − | :'''UPDATE:''' As of 9:30 am, the updates are complete and all ugrad services should be available again.
| |
| | | | |
| | :'''DETAILS:''' | | :'''DETAILS:''' |
| | | | |
| − | ::We need to update firmware on the CS Ugrad home directory file server. We'll be doing that starting at 8:30am on 6/29. We expect the update process to last up to two hours (as long as there are no issues.) Because the file server will be down, there will be several CS Ugrad computing services that will be unavailable during that time: | + | : Both the MH 122 '''and''' G61 Masters and Ugrad Labs will be used for an ACM Programming Contest and will therefore be '''''closed''''' to those not participating in the contest. |
| − | | |
| − | ::'''CS UGRAD COMPUTING SERVICES ''UNAVAILABLE'' DURING THE SERVER UPGRADE'''
| |
| − | | |
| − | ::* '''''None''''' of the CS Ugrad lab machines ''(ugrad1 through ugrad24, ugradx, ugradz)'' will work. They'll be running, but users won't be able to log in to them, and they'll appear to hang for users already logged in at the time the outage starts. These systems should return to normal after the outage is over.
| |
| − | | |
| − | ::* CS Ugrad web pages ''(under www.ugrad.cs.jhu.edu)'' will not load properly.
| |
| − | | |
| − | ::* Users with CS Ugrad email accounts will not be able to retrieve their mail. Sending mail will technically work, but if your mail client is configured to save copies of messages to a mail folder, that will not work (and might prevent the mail from being sent). Mail sent to CS Ugrad accounts will be available after the outage is over. (That incoming CS Ugrad mail will queue on our central mail server during the outage.)
| |
| − | | |
| − | ::* ''dbase.cs.jhu.edu'', the MySQL server used for the Databases class, will be unavailable. -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!-- -->
| |
| − | <!--:<span style="color:#ff0000">'''MAJOR CHANGE: The planned power outage for Malone Hall for Sunday, March 22, has been cancelled. Our systems will remain available for use.'''</span> ==>
| |
| − | <!--** PLANNED POWER OUTAGE TEMPLATE **
| |
| − | *******************************************
| |
| − | ********* POWER OUTAGE TEMPLATE***********
| |
| − | *******************************************
| |
| − | :'''WHAT:''' There will be '''''no''''' CS IT computing services available during an upcoming major power outage for ALL of Malone Hall.
| |
| − | | |
| − | :'''WHEN:''' Saturday, January 16, 2021, ''all day'', from 6''am'' until sometime after 8''pm''.
| |
| − | | |
| − | :'''DESCRIPTION:''' JHU is planning an all-day power outage for Malone Hall. As such, all of our computing services will be unavailable during that time.
| |
| − | | |
| − | :The actual power outage is planned to start at 7am and end at 8pm Sunday, January 16. However, we need to safely bring down our computer servers and clients earlier than that (before the power goes out.) We plan to bring our systems down around 6am that morning. We will then bring the systems back up after 8pm (when the power outage is declared over.)
| |
| − | | |
| − | :<span style="color:#ff0000">Please read all of the notes below regarding Malone Hall Residents, CS Server Room users, and a list of the CS IT services that will be unavailable during the outage. </span>
| |
| − | | |
| − | :''NOTES TO MALONE HALL RESIDENTS:'' During the time of the outage, we're assuming most building lights will be out, and the J-Card system might be out too, and that might make it more difficult to get into the building during the outage. But, with the power out, we're not sure how much need there will be to be in the building during that time anyway. If you have an office refrigerator, expect that to be out too. And, if you are counting on UPS (battery-backup) systems you have to power up your office equipment, please realize that this is an expected 13-hour outage. And, keep in mind that your office phones may be inoperable during the outage. So, please plan in advance for this outage.
| |
| − | | |
| − | :''NOTE TO CS SERVER ROOM (MH G-29) SERVER OWNERS:'' Our server room will be out of power, so our servers will be unavailable. If you host servers in the server room, your servers will be down as well. Plan ahead of time to shut down and to bring back up your servers.
| |
| − | | |
| − | :'''CS IT services unavailable during the power outage include:'''
| |
| − | | |
| − | :* CS DNS-based lookups (hostname lookups) will be unavailable.
| |
| − | | |
| − | :* CS Mail service, ''both incoming and outgoing'', will be suspended for CS Grad/Research Net users.
| |
| − | | |
| − | :* Incoming mail ''should'' be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − | | |
| − | :* IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable. And, although, CS Ugrads can send email through the JHU webserver, their @cs.jhu.edu incoming mail arrives at CS, so CS Ugrads might nMarch 22ot be able to see their email.)
| |
| − | | |
| − | :* No CS webpages will be available.
| |
| − | :**<span style="color:#ff0000">'''NOTE:''' This also means, '''no''' CS IT Support Wiki and '''no Notices page''' (this page!) We are looking into a temporary Notices page for that day, but at this time, no guarantees.</span>
| |
| − | | |
| − | :* Grad/Research Net and Ugrad Net user accounts and home directories will be inaccessible during the power outage.
| |
| − | | |
| − | :* SMB File Shares (the CS Office Staff uses these) will be unavailable.
| |
| − | | |
| − | :* ssh service to our CS Grad/Research Net & Ugrad Net Linux boxes (masters1 through masters16, ugrad1 through ugrad24, gradx, gradz, ugradx, and ugradz) will be unavailable.
| |
| − | | |
| − | :* Console (desktop) access at any of our lab Linux clients systems will be unavailable.
| |
| − | | |
| − | :* MySQL service via the mysql.cs.jhu.edu server and through the dbase.cs.jhu.edu server will be unavailable.
| |
| − | | |
| − | :* Network switches in the building will be without power, so no network connections in Malone Hall.
| |
| − | | |
| − | :* Wifi access points in Malone Hall will be without power, so no wifi.
| |
| − | | |
| − | :We'll post updates here as we get them, but probably not during the outage See the note about the Notices page, above.-->
| |
| − | <!-- TEMPLATES -->
| |
| − | <!-- TEMPLATE 1 START
| |
| − | | |
| − | | |
| − | <hr>
| |
| − | :'''WHAT:''' Linux kernel bug causing some of our Fedora 31 lab systems to hang.
| |
| − | | |
| − | :'''WHEN:''' First identified in CS, last week of January, 2020. '''''UPDATES below the DETAILS section...'''''
| |
| − | | |
| − | :'''DETAILS:'''
| |
| − | | |
| − | ::As you know, recently, we upgraded our lab Linux clients to Fedora 31. Over the past several days, we've discovered some odd behavior from some of the lab systems. Upon further investigation, we have learned that there is a kernel bug that is causing some systems to hang (requiring a reboot) when certain NFS operations are performed. (NFS is the system that allows our Linux clients to access the home directories, certain apps, other shared file systems.)
| |
| − | | |
| − | ::We're seeing that the bug is documented and that it affects not only Fedora 31, but some prior versions as well. It just happens to be that we are seeing its effects now.
| |
| − | | |
| − | ::This is a Linux kernel issue and there are Linux folk working on trying to get this bug patcMarch 22hed, but we do not know how long it will take.
| |
| − | | |
| − | ::In the meantime, CS IT is looking into work-arounds to bypass this bug as much as possible.
| |
| − | | |
| − | ::There is no way to predict if or when this bug will cause a system to hang. Some of our systems may never notice it. We think it might happen to some systems that are heavily utilizing NFS operations, but again, we're not sure.
| |
| − | | |
| − | ::Our recommendation is that when you're editing files, etc. save your work on a regular basis, just in case. This is a good practice to follow anyway.
| |
| − | | |
| − | ::'''''IMPORTANT:''''' If you experience a system that hangs (and doesn't recover after several minutes), please email {{mailto|support}} to let us know the name of the system that experienced the hang, the time of the hang, and what activity you were doing on the system at that time (which will help us troubleshoot the problem further.)
| |
| − |
| |
| − | ::We will update here with our progress on circumventing this bug.
| |
| − | | |
| − | ::'''UPDATE: 1/29/20''' We are applying alternative pre-bug kernels to those Linux clients that are available to be rebooted (to activate these kernels.) As more become available, we'll reboot them as well. We're hoping these alternative kernels will prevent the hangups until officially patched kernels become available.
| |
| − | | |
| − | ::'''UPDATE: 2/7/20''' We have applied updates to most of our systems in order to activate the alternative pre-bug kernels, which we hope will prevent hangups. There are still some systems which have not yet had the alternative kernels applied, and that will happen as the systems become available for us to reboot.
| |
| − | | |
| − | TEMPLATE 1 END-->
| |
| − | <!-- TEMPLATE 2 START
| |
| − | | |
| − | | |
| − | ===Suspended (Unavailable) ''Grad/Research Net'' services while rtfm.cs.jhu.edu & blaze.cs.jhu.edu are down:===
| |
| − | | |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − | | |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − | | |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − | | |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − | | |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − | | |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − | | |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − | | |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − | | |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − | | |
| − | | |
| − | | |
| − | | |
| − | TEMPLATE 2 ends -->
| |
| − | <!--:None at this time.-->
| |
| − | <!-- -->
| |
| − | <!--******************MORE TEMPLATES BEGIN HERE*****************-->
| |
| − | <!-- -->
| |
| − | <!-- ******TEMPLATE GRAD NET OUTAGE*****
| |
| − | ====Major CS Grad/Research Net service outage (mail, web, home directories, login, etc.) ====
| |
| − | <!----------------:None at this time.------------------------>
| |
| − | <!--;'''WHAT'''
| |
| − | :Major CS Grad/Research Net (and some Ugrad) IT services outage, as we update our file server system.
| |
| − | ;'''WHEN'''
| |
| − | :''Wednesday, June 13, from 5:30pm to approx 7:30pm.''
| |
| − | ;'''UPDATE'''
| |
| − | :June 13, 8:30pm. The service outage is complete, and are services our back to normal operation.
| |
| − | ;'''DETAILS'''
| |
| − | :Our CS Grad/Research Net home directory file server-related system, ''rtfm.cs.jhu.edu'', will be shut down so that important server-related updates can be applied. (In order to shut down that server, we must also shut down our mail, DNS, and other services server, ''blaze.cs.jhu.edu''.) We will apply the required updates on Wednesday, June 13, from 5:30pm to approximately 7:30pm.
| |
| − | :
| |
| − | :During the process of applying updates, several of our CS services will be unavailable. If there are any unexpected complications during the update process that will delay the restoration of services, we will update this CS IT Support Wiki's Notices page.
| |
| − | | |
| − | ===Suspended (Unavailable) ''Grad/Research Net'' services while rtfm.cs.jhu.edu & blaze.cs.jhu.edu are down:===
| |
| − | | |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − | | |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − | | |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − | | |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − | | |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − | | |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − | | |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − | | |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − | | |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − | | |
| − | : '''Note to Ugrad Net users:''' During the rtfm.cs.jhu.edu and blaze.cs.jhu.edu updates, email directed to your ugrad acct @cs.jhu.edu will not be immediately delivered. Other Ugrad Net services should remain available. -->
| |
| − | <!-- -->
| |
| − | <!--******************TEMPLATES BEGIN HERE*****************-->
| |
| − | <!-- -->
| |
| − | <!-- ******TEMPLATE GRAD NET OUTAGE*****
| |
| − | ====Major CS Grad/Research Net service outage (mail, web, home directories, login, etc.) ====
| |
| − | | |
| − | '''WHAT:''' Major CS Grad/Research Net (and some Ugrad) IT services outage, as we apply security updates to our systems.
| |
| − | | |
| − | '''WHEN:''' '''''Rescheduled for''''' Thursday, January 18, from 5am to approx 7am.
| |
| − | | |
| − | '''DETAILS:''' Many CS Grad/Research Net servers will be shut down to apply security updates. We will apply these updates on Thursday, January 18, from 5am to approximately 7am.
| |
| − | | |
| − | During the process of applying updates, several of our CS services will be unavailable. If there are any unexpected complications during the update process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at https://support.cs.jhu.edu/wiki/Notices
| |
| − | | |
| − | Suspended (Unavailable) Grad/Research Net services while the servers are updated:
| |
| − | | |
| − | * CS Mail service, both incoming and outgoing, will be suspended for CS Grad/Research Net users.
| |
| − | | |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − | | |
| − | * IMAP, POP, and Webmail from our CS (Grad/Research Net) Mail server will be unavailable.
| |
| − | | |
| − | * Some CS webpages, including user (and possibly lab and other CS pages), will be unavailable.
| |
| − | | |
| − | * Grad/Research Net user accounts/home directories will be inaccessible during the downtime.
| |
| − | | |
| − | * SMB File Shares (CS Office Staff uses these) will be unavailable.
| |
| − | | |
| − | * ssh service to our CS Grad/Research Net Linux boxes (masters1 through masters16, gradx, and gradz) will be unavailable.
| |
| − | | |
| − | * Console (desktop) access to masters1 through masters16 will be unavailable.
| |
| − | | |
| − | * MySQL service via the mysql.cs.jhu.edu server will be unavailable.
| |
| − | | |
| − | Note to Ugrad Net users: During the Grad Net servers' security updates process, email directed to your ugrad acct @cs.jhu.edu will not be immediately delivered. Other Ugrad Net services should remain available. Also, we will be applying security patches on the CS Ugrad net the next day, so you'll see an email for that at some point soon.
| |
| − | | |
| − | Updates, if necessary, will be posted here.
| |
| − | ******TEMPLATE END*****-->
| |
| − | <!-- -->
| |
| − | <!--******TEMPLATE BEGIN - UGRAD NET OUTAGE
| |
| − | | |
| − | ::::'''''NOTE: DATE CHANGE BELOW for OUTAGE #1!!!!!!!!''''' -->
| |
| − | <!------------==== Major CS ''UGrad Net'' service outage (mail, web, home directories, login, etc.) ====
| |
| − | | |
| − | | |
| − | '''WHAT:''' Major CS Ugrad Net IT services outage, as we reboot our Ugrad home directory file server.
| |
| − | | |
| − | '''WHEN:''' Friday, October 18 (Fall Break Day) from 9am to approx 10am.
| |
| − | | |
| − | '''UPDATE''': As of 9:45 am, all affected systems are back in service.
| |
| − | | |
| − | '''DETAILS:''' To patch a problem on our Ugrad home directory file server called barley, we need to reboot it. The boot process should take less than an hour.
| |
| − | | |
| − | During the reboot process, several of our CS services will be unavailable. If there are any unexpected complications during the reboot process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at https://support.cs.jhu.edu/wiki/Notices
| |
| − | | |
| − | Suspended (Unavailable) Ugrad Net services while barley is rebooted:
| |
| − | | |
| − | * CS Ugrad Mail service, both incoming and outgoing, will be suspended for CS Ugrad Net users.
| |
| − | | |
| − | * Incoming mail should be queued by the sender until our mail service is back up again. Outgoing mail will not be available.
| |
| − | | |
| − | * IMAP, POP, and Webmail from our CS (Ugrad Net) Mail server will be unavailable.
| |
| − | | |
| − | * CS Ugrad Net webpages will be unavailable.
| |
| − | | |
| − | * CS Ugrad Net user accounts/home directories will be inaccessible during the downtime.
| |
| − | | |
| − | * ssh service to our CS Ugrad Net Linux boxes (ugrad1 through ugrad24, ugradx, and ugradz) will be unavailable.
| |
| − | | |
| − | * Console (desktop) access to ugrad1 through ugrad24 will be unavailable.
| |
| − | | |
| − | * MySQL service via the dbase.cs.jhu.edu server (Dr. Yarowsky's class) will be unavailable.
| |
| − | | |
| − | Updates, if necessary, will be posted here. ------------------------>
| |
| − | <!--:'''WHAT:''' Emergency reboot of ''blaze'', the dept's main Grad/Research net mail server.
| |
| − | | |
| − | :'''WHEN:''' ''(TODAY!)'' Tuesday, Dec. 12 @ noon and lasting approx 1/2 hour if all goes well.
| |
| − | | |
| − | :'''DETAILS:'''
| |
| − | | |
| − | ::We have discovered an issue with our mail spool disk on ''blaze'' our Grad/Research net mail server which must be corrected right away. As such, we expect that a configuration change and reboot should fix the problem.
| |
| − | | |
| − | ::We plan to do this shortly, at ''noon today (lunchtime.)'' The reboot should take approx. 1/2 hour if all goes well. If there are issues, it might take some additional time to complete the repair.
| |
| − |
| |
| − | ::During the reboot time, there will be ''no CS email service in or out of the CS Grad/Research network.'' You will not be able to send or receive mail (IMAP/POP/etc.) through blaze (smtp.cs.jhu.edu/pop.cs.jhu.edu) or via webmail during that time. Also, DNS might be slow or non-existent, depending upon your client's own configuration.
| |
| − | | |
| − | ::You also might experience slow connection times to www.cs.jhu.edu.
| |
| − | | |
| − | ::Grad Net Lab Linux systems (masters1 through masters16, gradx, and gradz) may experience a short delay as their NIS service connects to a different server.
| |
| − | | |
| − | ::Any updates and notice of completion will be presented through this NOTICES page today.
| |
| − | | |
| − | :'''''UPDATE:''''' 12/12/17 at 12:15pm
| |
| − | | |
| − | ::Reboot successful! Blaze is back in operation and services have been returned to normal operation.
| |
| − | -->
| |
| − | | |
| − | <!-- :'''WHAT:''' Brief CS network outage
| |
| − | | |
| − | :'''WHEN:''' ''Rescheduled for'' Thursday, July 20, from 9am to 9:30am.
| |
| − | | |
| − | :'''DETAILS:''' JHU Networking (in conjunction with JHU Network Security) will be moving the various CS Networks in Malone Hall from one firewall platform to another. This will ''now'' take place on Thursday, July 20 at 9am and should only last approximately 5 minutes or so. But, we're providing a 30 minute window in case JHU Networking needs to troubleshoot any issues.
| |
| | | | |
| − | :During the network outage, CS systems might not be seen by each other, and network traffic between CS and outside-of-CS may be halted. Many CS services will be unavailable when the network is unavailable. Again, JHU Networking expects this to be a brief network outage, so you might only experience a short system hang. | + | : Additionally, all of the labs' Linux computers (grad1 through grad16 and ugrad1 through ugrad24) will be unavailable during the time period referenced above. |
| | | | |
| − | We will update this Notices page in case there are delays with restoring any of our services.
| + | * During the lab closure times, compute servers, gradx.cs.jhu.edu, gradz.cs.jhu.edu, ugradx.cs.jhu.ed, and ugradz.cs.jhu.edu '''''will continue to be available remotely''''', via SSH. |
| | | | |
| | + | * We have to take down the lab systems late that Friday afternoon. At the end of the ACM contest that Saturday night, the coordinators of that contest will be turning our lab computer systems back on. Most should boot normally. But, if some do not, just trying using an alternate one in the lab. We'll be back in the Monday afterwards and can check on any systems that didn't boot normally. (And don't forget, as mentioned above our compute servers -- gradx, gradz, ugradx, ugradz -- will continue to be remotely available, even if the lab systems are not.) |
| | | | |
| | + | * The printers in MH 122 and MH G61 will remain online, but, please pick up your printouts outside of the contest hours. |
| | | | |
| − |
| |
| − | -->
| |
| − |
| |
| − | <!-- ****** FILE SERVER OUTAGE TEMPLATE ******
| |
| − |
| |
| − | ===Planned Outage 1: CS Ugrad Net File Server Upgrade (barley.cs.jhu.edu)===
| |
| − |
| |
| − |
| |
| − | :'''WHAT:''' Major <span style="color:#ff0000">CS *Undergrad* Net computing services outage</span>, as we upgrade our CS Undergrad Net home directory file server,''' barley.cs.jhu.edu'''
| |
| − |
| |
| − | :'''WHEN:''' Wednesday, January 18, from ''8am to approx 11am''.
| |
| − |
| |
| − | :'''DETAILS:''' Our CS Undergrad Net home directory file server, barley.cs.jhu.edu, will be shut down to perform an major upgrade to its operating system.
| |
| − |
| |
| − | :Due to this shutdown for an upgrade, several of our CS '''''Ugrad''''' Net services will be unavailable during this time. If there are any unexpected complications during the upgrade process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at [https://support.cs.jhu.edu/wiki/Notices https://support.cs.jhu.edu/wiki/Notices]
| |
| − |
| |
| − | ====Suspended (Unavailable) Ugrad Net services while ''barley.cs.jhu.edu'' is down====
| |
| − |
| |
| − | :::*'''CS Ugrad Mail service,''' both incoming and outgoing, will be suspended for CS Ugrad Net users (as this fileserver works in tandem with our CS Ugrad mail server.) Incoming mail should be queued until our Ugrad mail service is back up again. Outgoing mail from CS Ugrad Net users will not be available.
| |
| − |
| |
| − | :::*'''IMAP, POP, and Webmail''' from our CS Ugrad Mail server will be unavailable.
| |
| − |
| |
| − | :::*'''Webpages''' served from our CS Ugrad webserver will be unavailable.
| |
| − |
| |
| − | :::*'''Ugrad Net user accounts/home directories''' will be inaccessible during the downtime.
| |
| − |
| |
| − | :::*'''ssh service to our CS Ugrad Net Linux boxes''' (ugrad1 through ugrad24,and ugradx) will be unavailable.
| |
| − |
| |
| − | :::*'''Console (desktop) access to ugrad1 through ugrad24''' will be unavailable. However, although the CS Ugrad computers will be unavailable during the fileserver downtime, the CS Ugrad Labs will remain open.
| |
| − |
| |
| − | :::*'''MySQL service via the dbase.cs.jhu.edu server''' (used by a databases class) will be unavailable.
| |
| − |
| |
| − | :Future updates will be posted on the [https://support.cs.jhu.edu/wiki/Notices Notices] page of the CS IT Support Wiki
| |
| − |
| |
| − | ===Planned Outage 2: Databases Class MySQL Database Server ('''dbase.cs.jhu.edu''') Unavailability===
| |
| − |
| |
| − | :'''WHAT:''' The Databases class's MySQL database server, dbase.cs.jhu.edu, will be unavailable when the CS Ugrad Net's fileserver, barley.cs.jhu.edu, is being upgraded (see above.)
| |
| − |
| |
| − | :'''WHEN:''' Wednesday, January 18, from ''8am to approx 11am''.
| |
| − |
| |
| − | :'''DETAILS:''' dbase.cs.jhu.edu uses the CS Ugrad Net file server and will therefore be unavailable during the barley.cs.jhu.edu upgrade time.
| |
| − |
| |
| − | :Future updates will be posted on the [https://support.cs.jhu.edu/wiki/Notices Notices] page of the CS IT Support Wiki
| |
| − |
| |
| − |
| |
| − | ****** END FILE SERVER OUTAGE TEMPLATE ******----->
| |
| − | <!-- ------------------------------------------------------------- -->
| |
| − | <!-- **VM Server Shutdown TEMPLATE **
| |
| − |
| |
| − | '''WHAT:''' The dept's Virtual Machine will be shutdown.
| |
| − |
| |
| − | '''WHEN:''' Wednesday, January 11 from 6am to 3pm (or earlier)
| |
| − |
| |
| − | We need to do some work on the VM server and will be taking it down during the above date and time.
| |
| − |
| |
| − | As part of the shutdown process, we will suspend most or all of the VMs; some
| |
| − | VMs might have to be shutdown all the way instead. And when the VM server is back up and running again, we will restart or unsuspend all VMs. In all cases, VM owner intervention should not be needed; we can handle this remotely.
| |
| − |
| |
| − | After the VM server is up and running again, VM owners will receive an announcement (and updates will be posted here), and, we do ask that VM owners check to make sure their VMs are running as expected. Please email [mailto:support@cs.jhu.edu support@cs.jhu.edu] if there are issues.
| |
| − |
| |
| − | ** END VM Server Shutdown TEMPLATE** -->
| |
| − | <!-- **Begin Specific Client shutdown TEMPLATE**
| |
| − |
| |
| − | '''WHAT:''' ugradz.cs.jhu.edu (one of our CS Dept Linux Ugrad Net compute servers) will go down for a memory upgrade.
| |
| − |
| |
| − | '''WHEN:''' Thursday, April 6, from 9am to 11am.
| |
| − |
| |
| − | '''DETAILS:''' ugradz will be taken offline so that its memory can be upgraded. While ugradz is down for its memory upgrade, you can still access our other Ugrad Net Fedora 25
| |
| − | systems (ugrad1 through ugrad24 and ugradx.) Info about our Fedora systems can be found at: https://support.cs.jhu.edu/wiki/Category:Linux_Clients
| |
| − |
| |
| − | Please make sure you are logged off of ugradz before the upgrade time.
| |
| − |
| |
| − | Any updates will be posted here.
| |
| − |
| |
| − | **End Specific Client shutdown TEMPLATE** -->
| |
| − |
| |
| − | == <span style="color:#ff0000">CS COMPUTING LAB AVAILABILITY </span> ==
| |
| | <br> | | <br> |
| − | '''MH-122''' (CS Undergrad Lab/Collaboration room): As of 3/13/20, '''''closed''''' until further notice.
| |
| − | :All CS Linux systems in that room (ugrad1 through ugrad12) will continue to be available remotely, via ssh.
| |
| − |
| |
| − | '''G-61''' (CS Masters/Ugrad Labs): As of 3/17/20, '''''closed''''' until further notice.
| |
| − | :For the UGrad side, although that section is closed, the Linux systems in that room (ugrad13 through ugrad24) will continue to be available remotely, via ssh.
| |
| − |
| |
| − | :For the Masters side, although that section is closed, the Linux systems in that room (masters1 through masters16) will continue to be available remotely, via ssh.
| |
| − | <!--'''NOTE:''' Remember, all of the CS Linux systems, including the ones in the labs, will be unavailable during the March 22 power outage. Please see above.-->
| |
| − | <!-- *************** END ACTIVE NOTICES ********************-->
| |
| − | <!-- :None at this time. -->
| |
| − | <!--PROGRAMMING CONTEST TEMPLATE -->
| |
| − | <!-- === MH-122 ''and'' MH G-61 (both CS Ugrad labs & the Masters Lab) CLOSURE ON Sat., Nov. 9===
| |
| − |
| |
| − | '''WHAT:''' The 1st floor CS Ugrad lab, MH-122, and both lab sides of the CS Masters/Ugrad Lab, MH G-61, will be closed due to an ACM Programming Contest.
| |
| − |
| |
| − | '''WHEN:''' Saturday, November 9, from 9am - 7pm.
| |
| − |
| |
| − | '''ADDITIONAL INFO:''' Although these labs will be closed for the ACM Programming Contest, you may still access our Ugrad Linux systems remotely, via ssh. Please see:
| |
| − |
| |
| − | :: [https://support.cs.jhu.edu/wiki/Linux_Clients_on_the_CS_Undergrad_Net https://support.cs.jhu.edu/wiki/Linux_Clients_on_the_CS_Undergrad_Net]
| |
| | | | |
| − | ::: or
| + | == {{red|CS COMPUTING LAB AVAILABILITY}} == |
| | | | |
| − | :: [https://support.cs.jhu.edu/wiki/Linux_Clients_on_the_CS_Grad/Research_Net https://support.cs.jhu.edu/wiki/Linux_Clients_on_the_CS_Grad/Research_Net]
| |
| − |
| |
| − | :for a list of our remotely available Linux systems. -->
| |
| | | | |
| | | | |
| | + | <!-- ***** NORMAL POSTING MH 122****** --> |
| | + | :'''MH 122''' (CS Undergrad Lab/Collaboration room) is available for in-person access. |
| | + | :{{red|However, please see the info above regarding the mid-November lab closure and systems unavailability.}} |
| | <!-- | | <!-- |
| − | *
| + | :'''MH 122 ''UNAVAILABILITY''''' |
| − | *
| |
| − | * ********************* BEGIN TEMPLATES ********************
| |
| − | *
| |
| − | *
| |
| − | *
| |
| | --> | | --> |
| − | <!-- ************ LAB CLOSURE TEMPLATE - START ***************
| |
| − |
| |
| − | ==== '''PLANNED CLOSURE''''' ====
| |
| − |
| |
| − |
| |
| − | '''''WHERE:''''' Malone Hall G-61, both the Masters and Ugrad sides of the lab.
| |
| − |
| |
| − | '''''WHEN:''''' Monday, March 14 & Tuesday, March 15, from 6am to 12 Noon.
| |
| − |
| |
| − | '''''WHY:''''' Network cable installation. The lab will be closed those two
| |
| − | mornings as workers install additional network cable.
| |
| − |
| |
| − | =====NOTES:=====
| |
| − |
| |
| − | * During the lab closure times, the masters and ugrad Linux computers in G-61 will continue to be available remotely, via SSH.
| |
| − |
| |
| − | * Ugrad Lab, MH 122 will not be affected and will remain open during those times.
| |
| − |
| |
| − | ************* LAB CLOSURE TEMPLATE- END *************-->
| |
| − | <!-- ********** BARLEY SHUTDOWN TEMPLATE - START *************
| |
| − |
| |
| − |
| |
| − | ===Planned Outage 1: CS Ugrad Net File Server Repair (barley.cs.jhu.edu)===
| |
| − |
| |
| − | UPDATE: 10/4/16 7:15am All systems back in operation!
| |
| − |
| |
| − |
| |
| − | :'''WHAT:''' Major CS *Undergrad* Net IT services outage, as we make repairs to the CS Undergrad Net file server, '''barley.cs.jhu.edu'''
| |
| − |
| |
| − | :'''WHEN:''' Tuesday, October 4, from ''6am to approx 8am''.
| |
| − |
| |
| − | :'''DETAILS:''' Our CS Undergrad Net home directory file server, barley.cs.jhu.edu, will be shut down to swap out a motherboard, since its own motherboard is reporting errors.
| |
| − |
| |
| − | :Due to this shutdown for repair, several of our CS Ugrad Net services will be unavailable during this time. If there are any unexpected complications during the repair process that will delay the restoration of services, we will update the CS IT Support Wiki's Notices page at [https://support.cs.jhu.edu/wiki/Notices https://support.cs.jhu.edu/wiki/Notices]
| |
| − |
| |
| − | <span style="color:#ff0000">UPDATE: 10/4/16 7:15am All systems back in operation! </span>
| |
| − |
| |
| − | ====Suspended (Unavailable) Ugrad Net services while ''barley.cs.jhu.edu'' is down====
| |
| − |
| |
| − | :::*'''CS Ugrad Mail service,''' both incoming and outgoing, will be suspended for CS Ugrad Net users (as this fileserver works in tandem with our CS Ugrad mail server.) Incoming mail should be queued until our mail service is back up again. Outgoing mail from CS Ugrad Net users will not be available.
| |
| − |
| |
| − | :::*'''IMAP, POP, and Webmail''' from our CS Ugrad Mail server will be unavailable.
| |
| − |
| |
| − | :::*'''Webpages''' served from our CS Ugrad webserver will be unavailable.
| |
| − |
| |
| − | :::*'''Ugrad Net user accounts/home directories''' will be inaccessible during the downtime.
| |
| − |
| |
| − | :::*'''ssh service to our CS Ugrad Net Linux boxes''' (ugrad1 through ugrad24,and ugradx) will be unavailable.
| |
| − |
| |
| − | :::*'''Console (desktop) access to ugrad1 through ugrad24''' will be unavailable. However, although the CS Ugrad computers will be unavailable during the repair time, the CS Ugrad Labs will remain open.
| |
| − |
| |
| − | :::*'''MySQL service via the dbase.cs.jhu.edu server''' (used by a databases class) will be unavailable.
| |
| − |
| |
| − | :Future updates will be posted on the [https://support.cs.jhu.edu/wiki/Notices Notices] page of the CS IT Support Wiki
| |
| − |
| |
| − | <span style="color:#ff0000">UPDATE: 10/4/16 7:15am All systems back in operation! </span>
| |
| | | | |
| | | | |
| | + | <!-- ****** NORMAL POSTING G61 ********** --> |
| | + | :'''MH G61''' (CS Masters/Ugrad Labs) is available for in-person access. |
| | + | :{{red|However, please see the info above regarding the mid-November lab closure and systems unavailability.}} |
| | <br> | | <br> |
| | | | |
| − | ===Planned Outage 2: Databases Class MySQL Database Server ('''dbase.cs.jhu.edu''') Unavailability=== | + | == {{red|AUDIO/VISUAL (A/V) OUTAGES}} == |
| − | | + | :None at this time. |
| − | :'''WHAT:''' The Databases class's MySQL database server, dbase.cs.jhu.edu, will be unavailable when the CS Ugrad Net's fileserver, barley.cs.jhu.edu, is being repaired (see above.)
| |
| − | | |
| − | :'''WHEN:''' Tuesday, October 4, from ''6am to approx 8am''.
| |
| − | | |
| − | :'''DETAILS:''' dbase.cs.jhu.edu uses the CS Ugrad Net file server and will therefore be unavailable during the barley.cs.jhu.edu repair time.
| |
| − | | |
| − | :Future updates will be posted on the [https://support.cs.jhu.edu/wiki/Notices Notices] page of the CS IT Support Wiki
| |
| − | | |
| − | <span style="color:#ff0000">UPDATE: 10/4/16 7:15am All systems back in operation! </span>
| |
| − | | |
| − | | |
| − | <br>
| |
| − | | |
| − | ********** BARLEY SHUTDOWN TEMPLATE - END ************* -->
| |
| − | <!-- ******************* MINOR OUTAGE TEMPLATE BEGIN ***********************
| |
| − | | |
| − | '''Wed, July 30, 9am through noon:''' ''Entire Undergrad Net'' (clients and servers) will be down as we complete system maintenance and move some equipment to a different rack.
| |
| − | | |
| − | Additionally, the Ugrad Net web and mail service will be unavailable during that time. (Incoming ugrad net mail should queue up on another of our mail servers during that time, until the ugrad mail server is back up and running and transfers the mail.)
| |
| − | | |
| − | ************** MINOR OUTAGE TEMPLATE END ********************* -->
| |
| − | <!-- ************** MAJOR OUTAGE TEMPLATE BEGIN *********************
| |
| − | | |
| − | '''MAJOR''' motherboard repair of ''rtfm.cs.jhu.edu'', one of our main CS ''Grad/Research'' Net file system servers, will create up to (but hopefully, much shorter than) 6-hour planned downtime for the Grad/Research Network. Our hardware service/repair company will be on-site completing repairs.
| |
| − | | |
| − | '''Outage will begin at 12am (midnight), Thursday, 8/22, and will last until 6am (or earlier) Thursday.'''
| |
| − | | |
| − | During that time, the following services will be ''unavailable'':
| |
| − | | |
| − | * ''Email in or out on the Grad/Research Net will be disabled.'' (Connections to smtp.cs.jhu.edu and imap.cs.jhu.edu will fail during that time. However, incoming mail to CS should queue at an alternative server.)
| |
| − | | |
| − | * ''Email on the Undergrad Net will be disabled during this time'' (as CS Ugrad e-mail addresses are registered via a server on the CS Grad Net.)
| |
| − | | |
| − | * ''DNS will only be available if you've configured your computer with a secondary DNS server.'' If you don't have a secondary server in your computer's DNS defined, please add 128.220.2.7 You may notice a delay in DNS name resolution when falling back to the secondary server.
| |
| − | | |
| − | * ''Grad/Research Net user accounts/home directories will be inaccessible'' during the downtime.
| |
| − | | |
| − | * ''Access to any of our Grad/Research Net linux clients (including gradx and masters1 through masters16, as well as Dr. Smith's pllab machines that we maintain) will be unavailable''
| |
| − | | |
| − | * ''peregrine.cs.jhu.edu (our Solaris login server) will be unavailable''
| |
| − | | |
| − | * ''Our mysql server, mysql.cs.jhu.edu, will be unavailable.''
| |
| − | | |
| − | * ''Samba shares will be unavailable.''
| |
| − | | |
| − | * ''Your Grad/Research Net web pages will be unavailable.'' (The webserver is expected to be serving minimal pages, mostly notices, during that time.)
| |
| − | | |
| − | ******************** MAJOR OUTAGE TEMPLATE END ************************* -->
| |
| − | <!-- ********* JHU-WIDE OUTAGES TEMPLATE - START *******************
| |
| − | | |
| − | ==JHU-WIDE IT SERVICE OUTAGES==
| |
| − | | |
| − | <br>
| |
| − | | |
| − | '''8/24/16 3:30pm''' UPDATE: We are told that JHU has fixed their mail servers and that JHU mail is back in operation again.
| |
| − | | |
| − | | |
| − | '''8/24/16 1:30pm''' JHU's central mail servers are experiencing an issue that is causing mail bouncebacks for people not on JHU Exchange sending to people on JHU Exchange. An error might look like:
| |
| − |
| |
| − | :'''The reason for the problem: 5.1.2 - Bad destination host 'Multiple DNS'''
| |
| − | :'''queries were attempted and failed: DNS Hard Error looking up'''
| |
| − | :'''feint.johnshopkins.edu<http://feint.johnshopkins.edu/> (A): NXDomain'''
| |
| − | | |
| − | JHU is looking into the problem at their end. Note that although the CS email servers are working fine, those on CS sending to JHU Exchange users may experience the same issue as others outside of JHU Exchange sending to JHU Exchange.
| |
| − | | |
| − | <br>
| |
| − | *********** JHU-WIDE OUTAGES TEMPLATE - END*********** -->
| |
| − | <!-- **************** MAIL BOUNCES TEMPLATE - START ****************
| |
| − | | |
| − | | |
| − | | |
| − | ===Possible Email Bounces From Normally-Working Sites===
| |
| − | | |
| − | '''8/8/16''' We have been receiving reports of ''email bounces from people/organizations that normally work without problems.'' We think we know what is happening. We recently discovered that one of our CS student accounts had been compromised and was being used to send spam. Even though we disabled the account as soon as the compromise was discovered, several mail server reputation services had already classified us as a possible source of spam. As a result of that classification, some organizations have decided to temporarily reject email from our mail server. We can't predict ahead of time which organizations those are, since there are a variety of reputation services, and individual organizations differ in the ways they make use of reputation information.
| |
| − | | |
| − | This problem will fix itself automatically over the next few days, as the reputation services take note of the fact that we have stopped sending spam. There's no way to speed up this process, unfortunately.
| |
| − | | |
| − | This sort of problem doesn't often happen to us. But when it does, it causes headaches like the unexpected bounce messages some people have received. We will be implementing some new processes that we expect will allow us to react more quickly to situations like this, before they reach the point where other organizations stop accepting our mail.
| |
| − | | |
| − | We will update this page once the reputation scores have returned to normal.
| |
| − | <br>
| |
| − | | |
| − | | |
| − | **************** MAIL BOUNCES TEMPLATE - END -->
| |
| − | | |
| − | == <span style="color:#ff0000">AUDIO/VISUAL (A/V) OUTAGES</span> ==
| |
| − | :None at this time. | |
| | | | |
| | [[Category:All]] | | [[Category:All]] |
