Author Topic: The future of oldWeather  (Read 5263 times)

philip.brohan

  • Old Weather Team
  • Full Member
  • *****
  • Posts: 198
    • View Profile
The future of oldWeather
« on: November 30, 2016, 02:20:26 pm »
oldWeather Classic is getting close to complete. We've heard the requests to put more ships on it, but it's not really possible - it's obsolete code - we don't have anybody capable of updating it. I'd really like to get to 100% finished. Let's get it done and move on.

 So the question is what to move on to. oW has changed a lot over the last couple of years - it's now much less a large public project with thousands of contributors and more the work of a group of experts. We are still generating lots of good data, but we are doing it quite differently.

 The plan for the new oldWeather was to go back to our original way of operating, to build a system for the complicated, data-rich logbooks we are working on now, that is sufficiently fast and easy to use that lots of casual volunteers will contribute - that we can scale-up to get millions of logbooks done. I'm sorry to say that we have not managed to build such a thing - the new user interface is faster and better (mostly) than classic, but nothing like easy enough to tempt casual users, and it's turned out to be very difficult to make work technically - it's complicated, and that means bugs. Our new site does not work well enough to do what we need - to become widely popular despite the difficult logbooks.

oldWeather-whaling works for the free-format whaling logs, and we'll keep whaling going for the low-observation density logs - but we need another approach for the government ship logs with tables of observations - those are where the vast bulk of the records we need are.

 We'd like to keep you  - the expert users - working on the highest-value difficult logs. When classic completes we won't have a web interface for you, but do we really need one? The web interface was for the thousands of casual users - I suspect that what experts need is actually the forum. If instead we were to provide links to the logbook images at NARA, and agree a spreadsheet format for data entry, and coordinate who was working on what through the forum, we might actually be better off.

 Kevin and I and the mods have talked about this, and we want to try it. If you are thinking, 'how will we enter data', the answer is 'however you like', we've seen plenty of good ideas for alternative transcription methods over the years, this is our chance to try some of them. We plan to come up with a simple template for putting the data in (columns for pressure, temperature, date, events etc.), but you will be free to choose how to do the transcription. For those who want to be provided with a tool, Bob has agreed to share one he has made, and I'm hoping others of you will also make suggestions, so we can all use one of those, or just make up out own.

 At the moment this is just an idea - we don't know exactly how it will work, and we are relying (as always) on the collective skill and experience of the forum to make it work at all. Please add your comments and suggestions to this thread.

 Please also push on with classic - let's get it to 100%. I was checking the progress statistics just before writing this, and one line stood out:

  "Total weather observations: 4,008,038"

Craig

  • Shipherd
  • Hero Member
  • *****
  • Posts: 3356
    • View Profile
Re: The future of oldWeather
« Reply #1 on: November 30, 2016, 03:22:53 pm »
Thanks, Philip. I am very relieved. The spreadsheet option sounds great, especially when you allow flexibility of input formats.

I was reluctantly thinking of stopping OW when there are no more Classic ships available but now I will certainly keep going!

mapurves

  • Shipherd
  • Hero Member
  • *****
  • Posts: 1837
    • View Profile
Re: The future of oldWeather
« Reply #2 on: November 30, 2016, 03:26:53 pm »
I think this is the only way to go. I do worry about the small number of people currently engaged in this project. I have been tracking the number of active transcribers and the number of WRs done each week. A year ago we were averaging about 20 transcribers completing around 22,000 WRs per week. We have been averaging just 13 over the past few weeks, but the totals remain much the same. Sadly, the number of active transcribers has been slowly decreasing over time. We need a strategy to get this number up, and I cannot think of a way to do this.

When I wrote my box drawing script, I added an option to work via an Excel spreadsheet, which was Stuart's idea. For OW3, it was a bit slower than using the script without the spreadsheet. However, it would be quick and easy if it didn't have to input the data into the little boxes. Although I didn't try it, the system could use the Open Office Calc program for those who don't have Excel.

The advantage with such a system is that I was enhancing the image using the free Irfanview software. The pages were much easier to read, and I had sized the spreadsheet so that spreadsheet grid overlayed the image. It was easy to enter the data in the cells which were directly over the values on the sheet. You could also see the values on each side (up and down) of the current value, which was helpful in deciphering poor writing.

I would urge that the Event pages be accessed totally separate from the data pages. It might be that casual users would be interested in entering the Events, and perhaps software as is being developed for the Shakespeare project would be suitable for the Events (it must present pages in chronological order). Some of us are not interested in the Events, while others are (myself included). Getting the images directly would solve this problem. The "data" people could just work on data pages, while those of us that like doing both could do so, perhaps using a differently formated spreadsheet for Events. Casual users could access the Events using Zooniverse software.

Once again, I think this is the way to go, but I worry about the decreasing number of transcribers. 20,000 WRs per week gives us about one million per year.

Michael

Code: [Select]
Date        People        WRs
20150803        19      22426
20150810        19      22837
20150817        20      21252
20150824        20      22735
20150831        23      19703
20150907        22      21757
20150914        20      22011
20150921        18      22817
20150928        20      21673
20151005        17      27094
20151012        17      22957
20151019        19      28960
20151026        17      18204
20151102        20      27867
20151109        16      25334
20151116        18      21673
20151123        18      23666
20160523        13      10676
20160606        16      18233
20160926        16      20512
20161003        13      21129
20161009        15      17706
20161017        15      18582
20161024        14      17810
20161031        15      20633
20161107        13      20207
20161114        13      22207
20161121        14      24252
20161128        13      21941

Hanibal94

  • Shipherd
  • Hero Member
  • *****
  • Posts: 4444
  • Better to do it, than live with the fear of it.
    • View Profile
Re: The future of oldWeather
« Reply #3 on: November 30, 2016, 05:17:48 pm »
Sounds great, Philip! If this is implemented, I would still contribute after OW Classic is done.

I would urge that the Event pages be accessed totally separate from the data pages.

That's a great idea, but there are some ships where weather and events are on the same page (like Jamestown 1844). Just wanted to remind everyone.

Helen J

  • Editor and Shipherd
  • Hero Member
  • *****
  • Posts: 4704
    • View Profile
Re: The future of oldWeather
« Reply #4 on: November 30, 2016, 06:43:49 pm »
This sounds well worth trying - I don't have the technical expertise to set up a system, but if we can draw on the collective wisdom of those who do, it could keep the WRs coming in, which I'd like to be part of.  I'll watch with interest ....

Randi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 13171
    • View Profile
Re: The future of oldWeather
« Reply #5 on: November 30, 2016, 07:45:53 pm »
That's a great idea, but there are some ships where weather and events are on the same page (like Jamestown 1844). Just wanted to remind everyone.

Yes...Like the armed trawlers of the Greenland Patrol ;)





Quote
Location | Port Name | Name = Bluei West One, Greenland                              Date = 25/12/1942

Location | Observed | Latitude = 61 08 N | Longitude = 45 26 W
Location | Observed | Latitude = 61 08 N | Longitude = 45 26 W
Location | Observed | Latitude = 61 08 N | Longitude = 45 26 W

Hour |Wind Dir |Force |Bar Height |Ther Attached |Dry|Wet |Water |Weather Code |Cloud Code |Clear Sky |
1NNE128.967BCAC3
2NNE128.966BCAC3
3NNE128.965BCAC3
4NNE128.965BCAC3
5NNE128.974BCAC3
6NNE128.994BCAC3
7NNE129.024BCAC3
8NNE129.024BCAS3
9NNE129.047CAS7
10NNE229.088CAS7
11NNE129.108CAS7
12NNE129.1110CAS6
1NNE129.1012CAS6
2NE129.1212CAS7
3NE229.1512CAS9
4NE229.1612CAS9
5NE129.1812SOAS10
6NE129.2014SOAS10
7NE129.2012SOAS10
8NE129.1912SOAS10
9NE129.1812OAS10
10NE129.1811OAS10
11NE129.1411OAS10
12NE129.1411OAS10
                                       
                                    

Quote
                                 
1200 MIDNIGHT to 400 AM
          Moored to port bow of U.S.S. LARAMIE outside of dock, at BLUEI WEST ONE,
GREENLAND. Ships present: USS NORTH STAR, AMAROK, SANDPIPER 1:45 Church party
reported in.
                                          N F Neer jr.
                                         
400 AM to 800 AM.
          Moored as before. 630 Crew performed holiday routine.
                                          N F Neer jr.

800 AM to 1200 MERIDIAN.
          Moored as before. 1030 CAPT. DONAHUE, G.R., U.S.ARMY MEDICAL CORPS,
reported aboard for transportation to BLUEI WEST EIGHT. Made daily
inspection of magazine. Condition damp. 1130 Pumping water
from fuel oil tanks.                                               
                                          Ernst W. Gschwender, Q.M.2/c

1200 MERIDIAN to 400 PM.
          Moored as before. 1232 Completed removing water from fuel oil
tanks, having removed 1200 gallons of water. Draft fwd. 10 feet.
                                          Ernst W. Gschwender, Q.M.2/c

400 PM to 800 PM.
          Moored as before.
                                          N F Neer jr.

800 PM to 1200 MIDNIGHT
          Moored as before.
                                          N F Neer jr.


(Thanks to Hanibal94 and fellow shipherds ;))
« Last Edit: November 30, 2016, 07:47:46 pm by Randi »

AvastMH

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7409
    • View Profile
Re: The future of oldWeather
« Reply #6 on: November 30, 2016, 08:51:40 pm »
I think that this amazingly open door that allows us to carry on with our good work in the ways that we have developed will probably help draw new people in anyway. I couldn't even begin to think to offer interesting talks to Women's Institute Meetings or Life Boat Society Groups with what we had to draw people in to. I feel confident that we may now be able to target the types of folk who are very likely to show an interest with something that is useful and allows confidence to be built quickly.  :D
« Last Edit: November 30, 2016, 08:54:57 pm by AvastMH »

Caro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7442
  • Our end is Life. Put out to sea. Louis MacNeice
    • View Profile
Re: The future of oldWeather
« Reply #7 on: November 30, 2016, 08:52:57 pm »
I'm also optimistic about using this plan to keep the project going. It has to be worth a try. :)

Randi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 13171
    • View Profile
Re: The future of oldWeather
« Reply #8 on: November 30, 2016, 09:09:11 pm »
Perhaps we will become the model for other transcribing projects ;)

Thursday Next

  • Editor
  • Hero Member
  • *****
  • Posts: 2425
    • View Profile
Re: The future of oldWeather
« Reply #9 on: November 30, 2016, 09:09:21 pm »
I currently only work on the histories, but it looks as if that will not be long in coming to an end, at which point I had fondly imagined I might go back to regular Old Weather.  I may have been an experienced transcriber at one time, but that is certainly not the case at present.  Would I still be able to contribute in the new format, given that I would be back to novice status?

Randi

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 13171
    • View Profile
Re: The future of oldWeather
« Reply #10 on: November 30, 2016, 09:34:18 pm »
Definitely !

In fact, your feedback might help us develop an interface (or interfaces - sounds like there could be more than one!) that you find easy to use.

mapurves

  • Shipherd
  • Hero Member
  • *****
  • Posts: 1837
    • View Profile
Re: The future of oldWeather
« Reply #11 on: November 30, 2016, 11:00:38 pm »
That's a great idea, but there are some ships where weather and events are on the same page (like Jamestown 1844). Just wanted to remind everyone.

True, but the labelling of the images could help. For example:

For vessels with Events and Data on Separate pages:
  • USS_Randi_345_D.jpg
  • USS_Randi_345_E.jpg

For vessels with Events and Data on the same page:
  • HMS_Joan_345_0.jpg
  • HMS_Joan_345_1.jpg

studentforever

  • Editor
  • Hero Member
  • *****
  • Posts: 3357
    • View Profile
Re: The future of oldWeather
« Reply #12 on: November 30, 2016, 11:17:41 pm »
I'm in the same position as Thursday Next. I HATE drawing boxes so if I return to weather transcribing then I need something which doesn't involve drawing my own boxes. That probably precludes me from most events interfaces - they all involve the hated boxes. I was never the fastest transcriber but I chucked my ha'porth into the pot and would consider doing some more on a bearable interface when I've finished editing. My idea of heaven would be OW WW2 but I fear that wouldn't fit the current software in the Zooniverse. I also found the freedom of the old interface appealing. I could spend a lot of time one day then do very little for the rest of the week so although more organization might be required I don't think I would like to be tied to so many pages per day or per week but a modest page allocation be be completed in my own good time would fit in better with my life and be much more  acceptable.

jil

  • Editor
  • Hero Member
  • *****
  • Posts: 2635
    • View Profile
Re: The future of oldWeather
« Reply #13 on: November 30, 2016, 11:20:29 pm »
Philip's suggestion sounds like a great idea - I was starting to get a bit sad at the thought of the end of Classic OW.

A spreadsheet in a smaller window over just an image of the log page to extract the weather data should work well. You wouldn't need the surrounding bit with the map and the image of the ship so the log page would be larger and easier to read, even without zooming in.

Perhaps events should go into a text file of some sort rather than a spreadsheet or just be left for the editor to do. Personally I would say extract all the info you need for the weather and leave the events for the editing process, but then I can't stop myself reading the events just in case there's something interesting.  ::)

Do we need all the separate types of events - 'animals', 'refueling' etc. tabs? Is this data being used? On the forum we're reporting aurora for space weather and volcanic activity is being passed on, so anything else that's actually useful could be handled this way.

Do we get to keep the forum or would we have to use Talk?

leelaht

  • Hero Member
  • *****
  • Posts: 1913
    • View Profile
Re: The future of oldWeather
« Reply #14 on: November 30, 2016, 11:23:05 pm »
Keep me posted.  I can do spreadsheets, but am self-taught and not expert, so we'll see if I can keep up.  :)

The forum is a highlight, please keep it going!  ;D