By Rowan McAllister and Yarin Gal
This post offers guidance for the workshop organization process post-workshop-acceptance:
– how to organize a successful workshop
– some pitfalls to be aware of
A version of this post was communicated with this year’s NeurIPS workshop organizers, but we are sharing this post more widely in order to increase transparency about the typical workshop organization process; and to encourage diversitywithin workshop organizers’ teams, specifically to encourage junior organizer’s participation.
So your workshop was accepted, congratulations on putting together a successful proposal! Your first task is to meet with all your workshop co-organizers. The goal of the meeting is to finalize co-organizer role assignments and synchronize everyone’s consensus on the important dates in your workshop’s timeline.
Further, consider establishing regular meetings with your co-organizers; either fortnightly, monthly, or prior to events like sending speaker invitations and starting the review phase, keeping co-organizers involved. Without follow up meetings, the lead organizer might get stuck doing most of the work, unbeknownst to the other organizers.
Milestones
Once accepted, the workshop organizers will verify that your workshop is making progress towards basic milestones. Dates for future NeurIPS workshops will need adjusting. Here are some example milestones and dates for the NeurIPS 2023 organization process.
– August 15th: Website URL active, including:
– Submission link,
– Easily-found contact info, ideally top or bottom of the landing page
– URL added to the NeurIPS site
– October 27th: Mandatory accept/reject notification
– November 3rd: Import accepted paper list to NeurIPS
– November 10th: Author deadline for Poster Printing Service
– November 20th: Schedule inputted into Schedule Editor on NeurIPS site (schedule between 8:15am and 5:30pm, with the schedule editor link that can be seen after an organizer logs in to the neurips.cc site).
– November 29th: Mandatory Slideslive upload deadline, if deciding to do any pre-recording
Dates for you to decide
– call for papers date (we recommend as soon as possible following workshop acceptance)
– submission due date (we recommend September 29th for NeurIPS 2023)
– review dates
– emergency review dates
– decision notification date (no later than October 27th for NeurIPS 2023)
– camera ready due date
We recommend all deadlines be scheduled at 23:59 (11:59PM) “Anywhere on Earth” time zone (AoE) to reduce confusion. You may also wish to email your speakers, confirming that their workshop is accepted.
Call for papers:
Send out a call for papers as soon as you’ve updated your website with your workshop’s timeline, paper format, and submission portal. Post about your workshop ideally sometime Tuesday–Thursday so more people notice it, and then follow up periodically with a few more online reminder posts in the lead up to the deadline (in case people miss the first one).
Submission date:
The suggested submission date for workshop contributions is September 29th for NeurIPS 2023, but this is really up to you. Late submission deadlines help attract more papers and more recentpapers. Early submission deadlines allow more time for reviewing; and/or also allows you to pull your notification date earlier too, to give authors more time to organize their visas and plan for their trips.
Whatever you choose, consider setting your submission deadline to be after theconference’s author notification dateSeptember 22nd to attract conference rejections too (which hopefully improve when submitted to your workshop). Avoid setting deadlines that fall on a weekend or common holiday.
Notification date:
Make sure to set your notification date so authors have enough time to get visas. The latest possible dateto notify authors of your accept-or-reject decision of their paper is October 27th for NeurIPS 2023. However, consider notifying authors of your decision before the Early Registration DeadlineOctober 21st to reduce your authors’ registration costs. Soon after the notification date you’ll also want to decide on contributed or spotlight talks (if any) so authors have time to prepare.
Outreach:
Schedule a set of announcements and reminders about your workshop using social media to remind people about your workshop beyond a one-and-done call for papers that people might forget about. Consider assigning the role of social outreach to one of your co-organizers so you know it will get done reliably.
Keep Speakers Updated
It’s important to keep speakers updated with relevant developments (like when your workshop is selected, and when the conference selects which date your workshop will be). Writing update emails to speakers can be a good excuse to remind them of your call for papers (if it’s been called yet) and invite them or their lab to submit works too.
Speaker Cancellations
Speaker cancellations are normal. When faced with a cancellation, you can either replace the speaker with another speaker or not. Opting for fewer speakers means you can expand the interactive elements of your schedule; like posters (ideally 2+ hours), panel discussions (45+ mins), or each speaker’s question time (10 mins). So if you have an excessive number of speakers left (9+), consider using the time for more attendee interaction.
Call for Papers
There are multiple ways to call on the community to submit papers to your workshop:
– Social Media
– Adding Images with social media posts will make them more noticeable. You could use a graphic that represents your workshop’s theme or an image of your confirmed speakers with the session title.
– Tag your speakers and co-organizers, they often want to share and promote it.
– Tag NeurIPS in your call for papers post using the hashtag #NeurIPS2023or handle @NeurIPSConf.
– Mailing lists
– Institutions: Share with your lab, university department or company newsletter and invite participation.
– Individuals you know would be interested.
Paper Format
Consider copying the NeurIPS style files, editing slightly with a footer that says it’s your workshop. Please be clear to your authors about what you expect, such as additionally stating “we welcome papers up to 9 pages (max) not including references or appendix, as a single PDF”.
Extended abstracts:
Whatever format you choose, consider allowing 4-page submissions (“extended abstracts”). This is because some computer vision conferences consider peer-reviewed workshop papers exceeding4 pages as prior publications. Extended abstracts (4 pages or less) therefore allow authors to submit preliminary results to your workshop before a conference submission without later violating a dual/double submission policy, which are the types of papers you want: exciting up-and-coming ideas that might not even be on arXiv yet.
Submissions
Archival or non-archival?
An archival or formally published workshop proceedings often precludes authors from submitting an extended version of the same work to other venues such as a conference or journal. Some authors therefore ask about this, so organizers usually specify on the website that “no submission will be indexed nor have archival proceedings”. In some situations, you may invite submission for both proceedings and non-proceedings. For example, full-length papers go to proceedings, extended abstracts go to non-proceedings. Note that NeurIPS will not itself publish proceedings for workshops: workshops will need to set up their own proceedings if desired.
Reviewing
Reviewing submitted papers is optional. Some workshops provide reviews since feedback is helpful to authors, but not all workshops do. Some workshops prefer lightweight reviewing: largely checking for fit and correctness and a selection of stronger papers for longer presentations. Such lightweight reviewing is more lenient, only rejecting poor or off-topic submissions, corresponding to a very high acceptance rate. Detailed reviews are more valuable to authors but place more burden on reviewers and can be less practical for larger workshops to provide reviews of consistent quality. If you wish to provide reviews, consider recruiting a program committee(aka reviewers) if you expect 15+ papers, otherwise the organizers can review the papers themselves. The organizers can choose between single blind and double blind reviews considering additional workload and fairness.
Program Committee
Be courteous:
To invite people to help review, people are much more likely to respond to a personalized invite from a human rather than an automated email from your submission portal. So consider reaching out to people first to ask, and add them into a reviewing system once they accept and are expecting automated emails. When writing the email, explain what you’re trying to do, and ask if they’d like to help review. To keep the workload low, not more than 3 reviews per reviewer are expected. You can also cc your co-organizers on the email to increase the recognition each reviewer receives, though if sending many reviewer invites then consider scheduling the emails to send at the same specific time to avoid distracting your co-organizers with a slow trickle of emails.
How many reviewers?
Aim for 3 reviews per paper so that the majority of papers receive 2–3 reviews. That means inviting about as many reviewers as expected submissions. Tell them the reviewing dates, and follow up with 1–2 polite reminders as the due date approaches to those yet to review. If you do this, only a minority of submissions will receive 0–1 reviews. Many reviewers only do reviews close to the actual deadline, so if you schedule a couple days between the reviewing due date and the notification date, then you can give late reviewers a little more time if they ask for it.
Emergency reviewers:
In anticipation of some papers with 0–1 reviews, you can recruit some “emergency reviewers” in advance to be “on call” for the couple days prior to paper decisions, so that every paper can have at least 2 reviews by the time you made your acceptance/rejection decisions. And if all else fails, youreview the papers with fewer than 2 reviews.
Set expectations:
You want to attract novel ideas that are scientifically interesting, on topic, and would create interesting discussions at the workshop. No need to be super critical of experimental results. Workshop papers are not conference papers so communicate to reviewers the criteria you’re looking for.
Reviewer questions:
Reviewing shouldn’t be laborious for your program committee, so consider limiting the amount of long-answer fields and add instructions to each field. You can even add the reviewing rubric to your website to help answer submitters’ questions like “my paper contains ABC, but not XYZ, is this good enough for a submission?” Consider including a score on reviewer confidence too.
Paper matching:
To match reviewers to papers, several tools are available. OpenReview has its own matching mechanism. Proper matching benefits are nice for authors and reviewers alike, especially if your topic is broad, or is an application(not method) of AI. For small workshops, bidding is probably overkill and takes time, so you can also manually match reviewers, by looking up their google scholar to classify their expertise in a spreadsheet, then classify the papers into clusters of subtopics on the same spreadsheet, then match like-to-like. Systems like OpenReview help avoid certain conflicts of interest by preventing you from accidentally matching an author and reviewer from the same institution.
Decisions
You can set the bar for acceptance however you like. Generally workshop decision thresholds are more lenient than conferences, resulting in the acceptance of 50–90% of submissions. You can consider aiming for three reviews per paper, and then make decisions based on the three reviewer ratings as follows:
– No accept ratings = reject
– 1/3 accept ratings = investigate, read reviews thoroughly, organizer makes decision
– 2/3 accept ratings = probably accept (check review that rated submission as reject)
– 3/3 accept ratings = accept
Keep in mind some reviewers will review in “conference mode” as if these were conference papers, despite any expectations you set initially. Workshop papers are not conference papers (yet). You want to attract new, scientifically interesting ideas; even if the experimental results are premature, or don’t outperform the state of the art, or even compare to it yet. So if the idea is on-topic and novel, consider dismissing any reject ratings that were based on lackluster experimental results and comparisons.
Notification:
Once you’ve decided which papers to accept and reject, then notify the authors by your promised notification date (which should be on your website). If papers were reviewed, make sure the reviews are viewable by the authors and remind them of the link so they can use the feedback to improve their camera-ready paper if accepted or some other venue if rejected.
Before the Workshop
Checklist:
- Organizer presence: Schedule which organizers will attend the workshop when. Ensure you have 2+ organizers actively monitoring things at any one time concurrently.
- Organizer communication: Setup a private messaging system for communication between organizers, e.g., with slack, so that on the day of the workshop organizers can coordinate quickly resolving issues as they arise. You can also simply use the slack account that the NeurIPS workshop chairs have invited some of you to already and chat in your specific workshop channel.
- Calendar invite for speakers: Make it impossible for your speakers to be confused with locations and time zones by sending them each a calendar event for their talk. Include room+location for the event and the Zoom link if they are remote. This helps if they are running late, and don’t have time to find such info buried in their email somewhere. You could even remind your speakers traveling to New Orleans to set their time zone on the website to America/Chicago.
- Prepare introductionsfor each speaker (their bio) for a warm introduction and smooth transitions between speakers. If you are unsure how to pronounce their name, ask them privately in advance.
- Obtain any consent and release forms (permission to record) from all speakers (keynotes and author talks) supplied to you by any recording groups the conference uses (if any) like SlidesLive.
- Print messages: To stay on schedule, help speakers avoid going overtime by bringing boldly printed numbers that you can hold up to inform them how many minutes they have left. Pack a Sharpie!
- Print schedulesto pin up outside the room (so conference attendees walking outside can check when they might want to join).
- Spare stuff: Bring spare AV adaptors including Apple adaptors, since AV issues often arise on the day, and spares are useful. Also bring your laptop, USB drives; and spare tape and push pins for posters. Pack dongles for laptop to HDMI connections.
- Backup questions: Prepare “backup questions” for each keynote or spotlight speaker. When each speaker concludes their talk, you’ll invite questions from the audience, but sometimes they cannot think of anything to ask initially. In such cases, it’s great to have some questions pre prepared. You can prepare by watching speakers’ talks in advance if recorded, or reading their recent on-topic papers in advance.
- Attend at least one Workshop Organizer meeting hosted by conference support staff
- Enter schedule into the neurips.cc website
- Some workshops will be using new remote cameras. This does require that the workshops provide an organizer to select the stream using a SlidesLive stream box. Plan to identify this organizer to SlidesLive and they will need to attend a training session for the stream box (about 20-30 minutes). Recommended that more than one organizer attend this training so that the job can be shared.
- Identify any special needs for room setup several weeks before the conference. A service kit will be provided for additional hardware needed or special room sets. Be sure to pay attention to the deadline dates. i.e., extra mics, room set other than theater, sponsor materials, catering, etc.
Day of the Workshop
It is likely that something will go wrong on the day of your workshop. Redundancy helps here. So have multipleorganizers attending the workshop to deal with issues together. Carry a spare laptop, AV adapters, and USB drive in case speakers have equipment issues. Consider having a private schedule with the other organizers who will be the master of ceremonies (MC) when, and who will be watching on standby as “backup MC” ready to take over if anything goes wrong.
Checklist: arrive early and:
- Reserve front seatsfor organizers + speakers (staff office will have “reserved for speaker” seat signs for pick up).
- Greet audio-visual (AV) staff, if any, to understand how everything will work, and explain any signals they’ll be gesturing toward you from the back when you’re up at the front talking during the day. There’s often AV issues on the day (a speaker doesn’t have the right adapter etc) and the AV team is critical here, so it’s good to get to know how the AV team (fully-dedicated or rotating) would support you, and thank them at your workshop’s conclusion.
Poster Sessions
Attend your own workshop! Go chat to authors during the poster sessions and learn about their work. As an organizer, you are recommended to visit posters that are not receiving as much attention. People are social, and crowds assume social proofthat certain posters are better if other people are already there, creating unequal distributions of crowded posters and lonely posters. You can counteract this bias by visiting less populated posters, which will attract others.
Talks
Even if you prepared “backup questions” for each speaker in advance, for non-recorded talks, you’ll want at least one organizer (the MC) to be paying attention to each speaker during the event to think of backup questions live. If no one asks a question, then you can ask your backup question. Questions beget questions that will warm up the audience.
After the Workshop
After the event, there’s several options to consider:
- Thank any audio-visual staff: After the workshop, thank the audio-visual staff and consider adding their names to the website. They do much critical work behind the scenes.
- Get feedback, e.g., via a link on your website or email to improve next year (example).
- Dinner: You can organize a dinner for the speakers and (if you have them) sponsors. This is nice for the speakers, and a good opportunity to get to know them better. It can be a bit pricey to buy a dinner for ~10 people, but one of your sponsors may be happy to pay the bill. Have some time between the workshop end and dinner though, people often want a break, or want to chat to others after the workshop instead of rushing off to dinner straight away.
- Link recordings to the website. It’s nice for speakers to have public links to their talks and for others who couldn’t make your event to see. E.g., SlidesLive might make links public a month after the workshop, or if you have the Zoom recording, you could upload this to YouTube with time tags for different speakers (example).
- Summarize submissions: Some organizers like writing up a paper or blog or website to summarize the workshop, something easily spread on social media. This helps give further publicity to workshop papers and the workshop itself. Most workshops do not take this extra optional step, but this could set the workshop apart.
You’ll need a webpage, to summarize the speakers, schedule, and how to submit papers.
Accessibility:
Not all websites are accessible to everyone. Google Sites is very easy to use, but not accessible in China. GitHub Pages is a little harder to use, but accessible in more countries.
– Custom URLs: Alternatively, you can provide custom URLs to make certain sites accessible. For example https://www.icra2022av.org (accessible in China, and since expired) was actually this website under the hood https://sites.google.com/view/icra2022av/home (inaccessible in China), so the workshop organizers just communicated the .org domain version. You can do this by going to your google site, then settings→ custom domains → add → buy a domain (~$12 a year).
– Github or Google Sites? An advantage of google sites: quick to create + upkeep, for anyone in a team. Advantage of github: get more control on appearance and enable javascript. For example, people can get confused with time zones, so you can always use the “Anywhere on Earth” (AoE) time zone for deadlines, and include a [javascript countdown](#submission-countdown) so people know exactly when the deadline is, or linking to https://time.is/Anywhere_on_Earth. For github you can use this template which looks like this initially, which you can change to look like this. Or use GitHub Pages which is simpler, looks like this; or http://jekyllrb.com/ which looks like this. An advantage of github is you can upload and host papers directly on your website.
– Other: You could also host and manage websites using Amazon S3 buckets
Helpful Tips / FAQ:
You can add suggestions where authors can print their posters near the venue (easier for one person googling this than 50). Some workshops even include poster templates.
Workshop sponsorship is optional. You do not need sponsorship. Some workshops use sponsorship to fund best paper awards (e.g., cash, cloud credits, or hardware). Although prize amounts not exceeding the cost of travel may not provide extra motivation to submit. Researchers are already motivated to submit papers and often best-papers are won by well-funded labs already. In our opinion, it’s better to fund student travel awards based on financial need, to enable more people to attend your workshop or at least reduce the financial stress of doing so (example application). It helps create a more equal research community and can increase physical attendance. However, workshop sponsorship can raise new challenges, for reasons below.
Warning: handling funds can be difficult.
If you have multiple sponsors, then holding funds together can be difficult. You generally cannot hold funds in your own personal bank account. You can open a new bank account, or you can ask a trusted third party like a university to hold funds for you, for a fee (e.g. Toronto might if you’re a student/staff there, Berkeley may not unless the event is on campus). The easiest option could be asking sponsors to transfer to recipients directly without involving third parties. When companies want to sponsor using their corporate credit card, use PayPal. PayPal is the easiest way to make international card transactions to individuals, and works in almost every country (Venmo is US only). If the sponsor requests an invoice for internal accounting purposes, an example is like this template. Please note that NeurIPS does nothandle sponsorship for your workshop. You need to handle it on your own.
Warning: hardware gifts can be difficult.
Certain countries unfortunately have export bans against other countries, especially for computing hardware. If you intend for a sponsor to give hardware to the “best paper award” authors as the prize, ask the sponsors if there are any constraints on authors’ nationalities. If constraints do exist in any country you’d expect to attend at NeurIPS, then you could consider not offering such a prize.
Other considerations.
Sponsors might also request proofthat prize recipients are in fact the competition or best-paper winners. Emails and websites might not count as proof, but event video recordings might. Find out in advance!
This post was adapted from https://rowanmcallister.github.io/post/workshops/ with the help of the original author Rowan McAllister. Original acknowledgements carry (Xinshuo Weng, Erin Grant, Andrea Bajcsy, Thomas Gilbert, Roberto Calandra, Yarin Gal, Pieter Abbeel, Wei-Lun Chao), and additional advice about pre-workshop acceptance is available in the original post.
