Category Archives: News

Game Dev Update: Twisty little passages, all alike

A quick update this time — I’ve been working here and there on implementing some maze generation code for the game, so that I could have a few different types of dungeon levels to generate.  At last I’ve got it working, and can now generate some intimidating mazes using the ‘growing tree’ algorithm:

maze16

By altering the likelihood of new branches in the path, I can change the feel of the maze significantly.  The maze above has a high likelihood of producing new branches; the one below produces much longer hallways:

maze19

 

Tonight I’ve just added a variation of this algorithm which mimics another well-known maze generation method, the ‘recursive backtracker’.  This one needs some fine-tuning, though, as currently it produces very long, meandering corridors that can be a little annoying to navigate:

maze20

The next step is to make the maze generator a bit more flexible.  Ultimately what I’d like to do is allow the normal dungeon generator to create maze rooms which can be integrated into the rest of the dungeon.  This will add some more variety in the dungeon without forcing the player to navigate an entire level-spanning maze every time the dungeon generator decides to mix things up.

I do think I want to have one dungeon level that’s entirely a maze, though, and encourage exploration by sticking a powerful artifact somewhere within and dropping some hints that the player might find it if they have a look around.  I’ll also scatter some Scrolls of Clairvoyance about, which will reveal the location of the level exit and make navigating the maze less directionless.

As you might’ve guessed, in these screenshots I’ve switched off the ‘fog of war’ for the player so that I could observe and test the results of the maze generator.  In actual play things look more like this:

maze17

By way of comparison, here’s how a maze level looks in the famous(ly difficult) roguelike Nethack:

Image result for nethack gehennom

More to come next time, when hopefully I’ll have maze rooms working.

 

Tagged , , ,

Game Dev Update: Upping the Challenge

So it’s been a little bit since the last update on my hobbyist game development efforts, but a lot’s been going on whenever I can snatch some time in between the constant, endless presentations I’ve had to prepare for recently on the academic side of my life.

My focus in the last few weeks has been on some core gameplay systems.  Originally I was going to work on the dungeon environment, but I figured that fancy dungeons wouldn’t be that useful if the player couldn’t do interesting stuff in them, so I went for under-the-hood gameplay systems instead:

Hunger system

Hunger systems are a classic feature of roguelikes — since the original Rogue, in fact!  The idea is that the player needs to seek out food within the dungeon and eat it regularly, otherwise they gradually begin to starve to death.  Since food is limited and can only be found within the dungeon and not created by the player, it serves as a time pressure mechanism; the player has to keep moving further down the dungeon to find food in order to stay alive.  Currently my system is very basic: the player starts with three rations, and can find two different types of food items within the dungeon that refills their hunger meter.  If at any point your Satiety stat dips below 50, you start getting warnings, and at 0 begin taking damage every turn.  At -50 you’ll die of starvation.  I’ve added an indicator to the interface that shows this stat directly, unlike Rogue and some other games that keep it hidden and only warn you shortly before death:

satiety7

Turn Scheduling:

In order to make monster fighting more interesting, I wanted to add a system for variable attack and movement speeds between different creatures.  This is kind of a weird thing to implement in a turn-based game, and I wasn’t sure of the best way to go about it at first.  Eventually I settled on a central turn scheduling system in which all monsters (and the player) schedule their next turn each time they take an action.  The number of turns they have to wait until they can move or attack again depends on their Speed stat.

It sounds simple, but it turned out to be a big pain to get right!  I had a lot of weird bugs where certain monster turns didn’t register, or the game would suddenly stop running in turn-based mode and let monsters run rampant while the player was unable to move, and all sorts of other problems.  Eventually I got all that ironed out, and finally had fast-moving wolves and bats, slow-moving zombies, etc… only to find that in certain circumstances the player would encounter invisible and invincible enemies after moving to a new dungeon level.

After a few days of annoyance I worked out the problem — monsters were dying in combat, but a reference to them was remaining in the turn schedule, meaning that memory was still being allocated for their monster data by the program.  In certain situations that meant the scheduler would find that reference, say to itself ‘hey this here says there should be an orc doing something’ and the player would end up fighting a ghostly remnant of a previously defeated enemy!

I was able to fix that relatively easily once I figured it out — this is why we keep backups, kids.  Also it was a useful reminder to be more careful about my coding practices in some parts of the game, so I did quite a few bits of refactoring of the code to prevent anything like that happening again.

Combat System Changes:

Again in service of making combat more interesting I made some significant changes to the combat system to better differentiate enemies.  My favourite series of Japanese RPG games is the Shin Megami Tensei series, which are known for being set in a dark demon-infested version of Tokyo and for being really difficult.  What I love most about these games is that the combat system encourages tactical thinking by having weapons and spells deal a number of different types of damage, which demons can be weak to, immune to, or resistant to depending on their nature.

I implemented something similar to this, where each monster has weaknesses to certain types of physical damage (in this game they’re called Phys for general slashing attacks, Blunt for hammers/clubs, Pierce for arrows and spears, alongside numerous magical damage types like Fire, Ice, Thunder, etc.).  Hitting a monster’s weakness will do double damage to it, while hitting if it’s resistant you’ll deal only half damage.  Some monsters are immune to certain types of damage entirely — Skeleton Warriors, for example, aren’t bothered at all about being poked by arrows given their total lack of fleshy bits.

The idea is that this will push the player to experiment with different weapons, spells and items in order to dispatch enemies more quickly — particularly when they reach later dungeon levels, where monsters start appearing in large swarms and have powerful attacks.  I want damage types to be a major focus for the player, and for good attack choices to have significant rewards (and for bad choices to have a big impact!).

To make things a little easier for the player, I’ve added a Look command so that you can see some details about monsters in visual range:

look-command

Other than these big changes, mostly I’ve been squashing bugs and adding bits and pieces of content.  At this point, the player can face about 300 different monsters, collect 50 different weapons and items, and visit 15 different dungeon levels.  As time goes on I’ll keep adding new monster types, then randomly-generated weapons and items, and finally some super-tough boss monsters including a final boss.  At that point I think I’ll be ready for a playable alpha release to get some gameplay feedback.

This week has marked the official start of the academic year, so there’s been a ton of things to do at the university — which means there hasn’t been much time for game development!  I’ve been keeping careful track of all my additions and to-do lists and such, so when I have time to get back to things I’ll remember what I’d planned to do next.

All in all it’s a hell of a lot of fun so far, even when I’m getting frustrated by weird bugs.  I’m definitely gaining some major Python proficiency thanks to all this, and it’s forced me to tighten up my coding practices and embrace proper version control.

After some more work on damage types and attack variety I’ll be modifying the game UI and sprucing up the general look of things, so hopefully I’ll have some more interesting screenshots next time 🙂

I’ll leave you now with some game recommendations:

SDL Rogue: a well-done Windows port of the original Rogue, with some nice additional options (including a graphical tile option).  Type ‘?’ to see all the commands (there’s a lot of them!).  The same site also has similar ports of other classic roguelikes, including Hack (the predecessor to Nethack), Larn, and Moria (predecessor to Angband).  Check ’em out.

DoomRL:  This is a fantastic free game that combines two of my favourite things — the original Doom (in my opinion the greatest computer game of all time, and one I still play regularly), and roguelikes.  It masterfully combines frantic Doom-style demon-blasting with turn-based roguelike gameplay and character progression, and to top it off has the original Doom music and sound effects, and some fantastic 2D graphical tiles.  Download it!

Some enterprising fan has developed a server for DoomRL, too, so you can see how you stack up against other players and even play in your browser (or via a Telnet client).  ASCII text mode only though!

Crypt of the NecroDancer:  This game kicks my butt all over town, but it’s incredibly creative and fun.  It’s a turn-based roguelike with a twist: the entire game world is tied to the rhythm of the music for the level, and making your moves along with the beat gives you various bonuses (and moving out of time leaves you open to attack!).  All the items and monsters play on this theme.  The graphics are incredibly charming, the music is by Danny Baranowsky (of Binding of Isaac and Super Meat Boy fame) and is absolutely fantastic, and it’s packed to the brim with cool features and secrets to unlock.

It’s also 50% off on Steam right now, so now’s a great time to take the plunge!

Tagged , ,

Diversions: Game Dev Update

-WARNING: MORE GAME NERD STUFF-

So in my evening hours I’ve been working away on the roguelike game development project.  For the most part I’ve been really pleased with how well it’s trucking along — I’ve seen a number of other roguelike projects out there using Libtcod that fizzled out way before adding this amount of content, so I feel confident I can finish this project.

Having said that, it’s requiring a ton of changes in my coding practices.  Even in just a couple of weeks of development time, my game has expanded to the point where every addition has a large number of potential interactions with all the other game components, so I’ve had to give in and embrace version control.  Every change is archived on my own machine, then on Google Drive, and finally uploaded, checked, and merged into the master code branch on GitHub.  I’ve never bothered to do this before, but with this project (where just adding a new potion can cause weird stuff like making monsters stop moving completely until I find some minor malfunction) I’ve found I need to make it as simple as possible to back up a step when something goes strange.

I’ve added a few major gameplay additions since my last post.  One of them appears right at the start of the game — you can now choose a character role when you start the game.  Fighters are focused on close-range combat, Knights carry a shield which gives them great survivability, Archers are great at long-distance but weak up close, and finally Wizards are incredibly fragile but start with powerful spells.  Here’s a wizard who’s just managed to turn a wolf to stone:

update5

You might notice that I’ve been experimenting with the ASCII visual presentation too — the plan is to have some colour schemes for each set of dungeon levels, and each set will have particular characteristics and terrain hazards too.  I’ve also changed the lighting slightly, heightening the contrast between explored and unexplored squares, and I’ve changed the font to a 16×16 version of Terminal that I think is a bit easier on the eyes.

Since I’ve introduced Wizards, I’ve also introduced the beginnings of a magic system — although there’s a lot left to do in this area.  I’ve followed the lead of other roguelikes and developed a system of wand items — these drop randomly in the dungeon or from certain enemies, and give you a limited number of uses of a specific spell.  I’ve added eight wands, each with two varieties, as a first test of the magic system and how it affects play.  Later I’ll be adding a system for Wizards to learn spells permanently, a resource system to restrict usage, and more detailed stats for players and monsters to make certain spells more effective against particular opponents.

I particularly enjoyed making the Petrification effect — it turns the monster into stone, of course, which then blocks visibility and movement.  So you can use it tactically to block off other monsters and restrict their movement or vision.  When you tire of it you can destroy the statue and it crumbles into rocks (which you’ll be able to use as ammo for a sling, once I make that).

As for monsters, I’ve added a few of those, but more importantly I’ve added a random mutation system for monster creation.  When a monster is generated, there’s a small chance that the monster will be mutated either one or two times, and each mutation will add statistical bonuses and special abilities, as well as alter the monster’s name and display colour.  I’ve only added six mutators so far, but that’s already enough to significantly increase the game variety.  Honestly I probably spend more time play-testing than I really should when I have coding time — but I think that’s a good sign if I’m already finding it fun to play!  The mutators will be even better once I add more monster types — these are taking time as each monster type has it’s own AI function which takes a lot of testing to get right.

Here’s a Fighter who’s run into a Hellfire Troll, a significantly stronger mutation of the already difficult Troll:

update1

All told it’s been a productive week.  My goal for the weekend is to jazz up the dungeon environment significantly, first by setting up the visual themes and names for each level subset, then developing a system for adding terrain features, some of which will be interactive (think stuff like lakes, lava flows, grass, fungus, rock pillars, etc.).

After that there’s some major tasks in my development plan:

  • Flesh out the magic system (which will need a bunch of interrelated items and systems to go with it)
  • AI and game systems for allied characters (in some circumstances players will be able to raise some troops, by resurrecting dead monsters or brainwashing living ones)
  • Methods for random generation of items

Again none of these are new ideas — my design goal here is simply to make a complete roguelike in the classic style, with just a bit more playability and transparency.  Once it’s done I’ve got some ideas for game #2 — but I’m not allowing myself to think much about that now!

Speaking of roguelikes in the classic style, thanks to some dedicated roguelike fans out there you can actually still play the original Rogue on modern computers.  It’s still quite playable, albeit mercilessly hard — winning Rogue is a real accomplishment.

You can also still play Moria — the seminal early roguelike based on Tolkien which later spawned Angband.  Really though I’d recommend trying Angband if you want to try a classic roguelike — it’s much more user-friendly, has lots more content, and has a graphical mode.

Tagged ,

A Diversion: A First Attempt at Game Development

–WARNING:  LOTS OF WORDS INCOMING–

As my friends and colleagues are aware, my favourite hobby outside the academic sphere is gaming.  In particular I have a fondness for very difficult games that require some tactical thinking to survive — so I’m a big fan of roguelikes.

For those of you who don’t know what a roguelike is, I’ll share with you a too-lengthy post I made on Facebook by way of explanation:

My game is a ‘roguelike’, a genre named for a very popular game in this style from 1980 called Rogue. There’s some debate about how to define roguelikes precisely, but generally they’re characterised by:


1) Turn-based gameplay — you take a turn, then the enemies. You can take as long as you like to think about each action and its consequences.


2) Randomly-generated levels — often you dive into a very long, multi-level dungeon full of monsters, items and fiendish traps, and every level is randomly generated each time you play, meaning you never run out of levels to play and no play-through is ever the same. In my game dungeons are generated using BSP trees which is a pretty common method.


3) Permanent death — if your character dies, that’s it, you have to start over from the beginning! What’s more, most games (including mine) automatically save the entire game state after every move, and overwrite your saved game when you reload a save, meaning that you can’t ever go back to a previous turn. Every action has permanent consequences.


4) Text-based graphics are usually a feature of every roguelike in the classic style, even today in 2016. Graphical modes are normally an option nowadays too, but many people prefer text mode (including me) because once you get used to it the text actually makes it much easier to determine exactly what’s in front of you at any given time, which is important because…


5) …while the graphics are often simple, the gameplay is normally quite complicated. Hundreds of items, monsters, and environmental features are in these games, all of which can interact in tons of different ways — not needing to make fancy animations for everything means developers can go nuts with the gameplay systems.

Anyway those are the main features you see in the genre — most of the major games in the genre are completely free, so I recommend trying them. The site RogueBasin has links to the five ‘major roguelikes’ right on the front page — ADOM, Angband, Crawl, NetHack and ToME. In my opinion Crawl or ToME are probably the easiest places to start, given they have pretty nice visuals and decent tutorials for new players. Angband is a long-time classic and has a decent graphical mode included as well. NetHack is famous for having insanely detailed game systems and item interactions, but is a bit overwhelming for newcomers.

My game is very early in development, although the core gameplay systems are all functional (random dungeons, character progression, combat, item and magic systems) and it’s fully playable. Not bad IMO given I only started making it five days ago, but there’s a long way to go yet. My goal is to add at least one new gameplay feature, monster, item, etc. every day for the next couple months so I can gradually build up a complex but balanced gameplay experience.

Given that this is my first-ever foray into making a complete game — I don’t count my failed attempts to write code for the GameBoy Advance back in 2005 — I’m pretty pleased with my progress, and it’s certainly been an enjoyable and challenging way to spend a few days of my summer holiday.

I started off by following the Complete Roguelike Tutorial using Python 2.7 with Libtcod — this gets you a complete, playable prototype (albeit very simple).  Since then I’ve added quite a few major features, each of which has required learning a bunch of new algorithms or techniques:

  • A* pathfinding for monsters
  • Random dungeon generation via BSP trees
  • A graphical version (using free 16×16 tiles made for Angband)
  • A more complex combat system including dicerolls and critical hits
  • Damage-over-time (poison attacks)
  • A complete ranged combat system including ammo tracking and relevant attributes for the player and enemies
  • Several new AI routines designed for specific enemies, i.e.:
    • Wolves that call for their pack mates when in trouble
    • Snakes that stay at a distance and spit poison
    • Enemy archers who try to get the drop on you

Now that the core gameplay systems are mostly there, and the return to academic life is looming, I’ll be slowing down significantly from here.  My goal is to build on this foundation bit by bit over the coming weeks and months, until I have a cohesive 20-floor dungeon-crawl experience with quite a few dozen monsters, items and weapons to discover.  This is a bit of an experiment for me, and I may not release this game to the larger world and instead bank this experience for a better follow-up project.  Having said that, once it’s in a more complete state I’d welcome any interested play-testers!

I should note that if I do end up releasing this it’ll be completely free, and probably open-source, assuming my code isn’t too embarrassing.

Since this is ostensibly an academic blog I should say that part of what pushed me to finally take the plunge and try to make something like this was actually some of the students I advised last academic year.  A large proportion of my advisees were game development students, and I wished I’d had more domain-specific knowledge to help them through certain problems.  Now at least I can tell them where to look when I hear from students who are stuck on pathfinding, AI issues, etc.!

Before I leave you, a screenshot of the fabulous ASCII graphics I’ve been staring at for hours on end:

poison_screenshot9-cropped

And for the ASCII-shy, a screenshot of the graphical version:

tiles_screenshot6_cropped

More to come later — the graphical version is very much a side-project at the moment, so expect that to be spruced up as time goes on.  I’m planning to switch to the stylishly lo-fi tiles of Oryx Design Lab — they’re simple, visually clear, and as they’re uncoloured I can tint them a million ways to represent the requisite dozens of variations of every imaginable monster that roguelikes generally have.

If you think I’m exaggerating — NetHack variant Slash ‘Em Extended boasts 12,221 monster species.  I think I’ll stop long before the 12,000 mark though.

That’s enough out of me for today — I’ll post updates on here every so often, if I find any particularly interesting techniques or come up with a version polished enough to distribute.

 

 

Tagged , ,

Alife XV Presentation

I’ve been attending Alife XV all week in extremely hot and sweaty Cancun, Mexico.  Yesterday I gave a talk on my paper with Nic Geard and Ian Wood titled Job Insecurity in Academic Research Employment: An Agent-Based Model.

I really enjoyed giving the talk — I spent a great deal of time beforehand thinking about how to introduce the work in proper context, and in the end I felt it worked reasonably well.  I had some great questions which raised important points that we’ll be taking into account in the next iteration of the model.  I’ve had a number of colleagues share their enthusiasm about the topic since the talk, so I’m really pleased and hopeful this work will keep advancing.

Thinking about the feedback I received, I think the most important next step is to develop the competitive funding aspects of the model in more detail:

  • Instead of an optimistic world with research funding that scales with population, have a pot of funding which grows at a slower rate, leading to a gradually more selective competitive process
  • Test possible implementations of more varied grants — larger/smaller grants which can produce more postdocs, grants of a longer duration, etc.
  • Possibly too ambitious for the near future, but implementing a system of teaching quality/student funding which also requires time allocation from the agents would be an interesting direction to take this

I’ve uploaded the presentation slides, and the final published paper is available here.  The full Alife XV Proceedings volume is available open-access via MIT Press.

Tagged , , ,

Funded PhD opportunity at Teesside

Fully-funded PhD opportunity available! I’m looking for someone interested in working on agent-based modelling for healthcare applications. No fees and £20K stipend. These are four-year positions and you will be asked to contribute up to six hours per week of teaching (tutorials/demonstration only, no lectures), which is more work but also good for the CV. Click here and filter under ‘Computer Science’ to see my project.  For more about me, check out the various pages on this blog or my staff profile at Teesside.

Project description: This research will focus on the application of Agent-Based Modelling techniques to human social systems, with particular emphasis on digital health applications. In the context of public health, agent-based models can help us understand the complexities of health policy implementation and service delivery by modelling the multiple interacting processes underlying the health system. These models will investigate challenges in health and social care service delivery across a variety of spatial and temporal scales – from short-term studies of demands on accident and emergency services, to longer-term explorations of the pressures facing social care over the next several decades. Our multi-disciplinary team will work with members of the School of Health and Social Care here at Teesside, along with external collaborators and stakeholders. The project would be suitable for a graduate with a background in Computer Science, Artificial Intelligence, Statistics or Complexity Science with an interest in Public Health/Healthcare applications.

ACADEMIC FRIENDS: Please tweet/share this as widely as you can!

Tagged , , , ,

Paper accepted to Alife XV

I’m pleased to say that the paper I’ve been going on about now for some time, titled Job Insecurity in Academic Research Employment: An Agent-Based Model, has been accepted to Alife XV in Cancun this summer.  I’m currently working on some revisions to the paper to account for some helpful suggestions from the reviewers — as soon as the final camera-ready preprint is available I’ll post it here and the usual places (ResearchGate, Academia.edu, etc.).

Hope to see some of you in sunny Mexico come July 🙂

Tagged , , , ,

Paper Submitted To Alife XV

I’m happy to report that I’ve recently submitted a first paper on the postdoc simulation I’ve been plugging on these pages for some time.  I’ve been working in collaboration with Nic Geard of the University of Melbourne and Ian Wood, my officemate at Teesside.

The submitted paper is titled Job Insecurity in Academic Research Employment: An Agent-Based Model.  Here’s the abstract:

This paper presents an agent-based model of fixed-term academic employment in a competitive research funding environment.  The goal of the model is to investigate the effects of job insecurity on research productivity.  Agents may be either established academics who may apply for grants, or postdoctoral researchers who are unable to apply for grants and experience hardship when reaching the end of their fixed-term contracts.  Results show that in general adding fixed-term postdocs to the system produces less total research output than adding half as many permanent academics.  An in-depth sensitivity analysis is performed across postdoc scenarios, and indicates that promoting more postdocs into permanent positions produces significant increases in research output.

The paper outlines our methodology for the model and analyses a number of different sets of scenarios.  Alongside the comparison to permanent academic hires mentioned above, we also look closely at unique aspects of the postdoc life cycle, such as the difficult transition into permanent employment and the stress induced by an impending redundancy.  For the sensitivity analysis we used a Gaussian process emulator, which allows us to gain some insight into the effects of some key model parameters.

The paper will be under review for the Alife XV conference very shortly, so I don’t want to pre-empt the conference by posting the full text here.  If — fingers crossed — it gets accepted, I’ll post a PDF as soon as it’s appropriate.  If you want a preview or are interested in collaborating on future versions of the model, please get in touch!

Tagged , , , , , ,

Science about Science: Does Promoting More Postdocs Help?

Just a brief one today — I’ve been playing with parameter settings on the funding/careers model, particularly the impact of postdoc promotions.  In the base scenario, postdocs (referred to as PDRs here: Post-Doctoral Researchers) have about a 15% chance of getting promoted to a permanent position.  Here’s a sample run at the base settings (which includes the mentoring bonus added last time):

r_mean_pdr2

I’ve finally worked out how to fix the legends on these graphs!  Now let’s compare to a scenario in which 50% of postdocs get promoted:

r_mean_pdr2

 

Note that the mean productivity of grant-holders (the green line) is overall a bit higher than in the 15% case.  The productivity of promoted postdocs (the orange line) also tracks higher over time than in the 15% scenario.

Now let’s try 100% promotion chance:

r_mean_pdr2

Here the productivity of grant-holders and promoted PDRs is higher than in either the 50% case or the 15% case.

So does this mean that promoting more postdocs is our ticket to a more productive research community?  Well, in this virtual academia it seems to help — but still we’re seeing a lower level of productivity than in the postdoc-free scenario.  Not to mention that there’s still quite a bit of statistical work here to be done to determine how significant these effects are — but it’s an interesting result from today’s work and one I hope to address in the paper, assuming that the analysis bears it out.

 

 

Tagged , , , ,

Science About Science: More Scenarios

Since my last post I’ve been doing a lot of work on cleaning up the simulation and adding some additional scenarios to the mix.  After some in-depth discussion with colleague Nic Geard, co-author of the 2010 academic funding model that inspired this work, we decided that a good starting point for this would be to compare a more basic growing population of permanent academics with a population that includes insecure postdocs.

So that led to me getting to work on re-working some things to allow for four possible scenarios:

  1. Core academic funding model as written by Nic and Jason
  2. Simple growing population of permanent academics
  3. Population which includes postdocs, in which research quality does not increase the chances of promotion for postdocs
  4. Population which includes postdocs, in which research quality does increase the chances of promotion for postdocs

This last scenario in particular is intended to investigate how things proceed if we have an optimistic view — we know exactly how good each postdoc is, and we hire only the very best 15% of the current crop during each iteration.  Those in favour of the current structure would most likely argue that competition for limited jobs allows the cream to rise to the top, so we need to investigate whether that assumption holds.

So for the purposes of this post, I’ve done a quick run of the sim for each of these scenarios.  Note that the previous model by Nic and Jason investigates the time-management aspect of grant applications much more deeply — right now I’m just focusing on the mean research output for different groups of academics under each scenario.

Scenario 1: Core Academic Funding Model

If you alter the parameter settings of my version of this model and turn off all my additions — growing populations, promotion mechanisms, and the postdoc system — you end up with a scenario that’s nearly identical to the original model by Nic and Jason.  The only major difference is that in my version the bonus in research quality given to grant-holders is 1.5 rather than 1.25.

So what we see is that grant-holders, as you might expect, have a massive advantage in terms of research productivity:r_mean

Grant-holders are sitting pretty at the top there, although their output fluctuates given that various researchers of differing levels of research talent are jumping in and out of the grant-holders club each semester.

(NB: I’m aware that the non-grant holders are invisible in this graph and the next — I’m working on it.  This is all a work-in-progress, it’ll get there in the end!)

Scenario 2: Growing Population

In the second scenario, I’ve added a mechanism which adds a few academics to the population each semester.  Their research quality and initial level of time investment into grant proposals is randomised.  As in the last post we’re living in a generous society here where research funding stays in step with the growing academic population — 30% of applicants are always funded, regardless of the population size.

Perhaps unsurprisingly, the results look nearly the same as in Scenario 1:

r_mean

Just like in Scenario 1, grant-holders do far, far better than the overall population, particularly those applicants whose grant applications have failed.

Scenario 3: Postdocs, Random Promotions

So now things start getting more bizarre.  In this scenario we introduce the postdoctoral system outlined in my previous posts.  Postdocs are added in proportion to the number of grants that have been funded in a given semester, with a bit of random variation to spice things up.  New postdocs are assigned contract lengths between 4 and 10 semesters.  For the first two semesters their research quality is lower to account for their adjustment period into a new post; similarly, their last two semesters also see a drop in quality due to the time they must devote to finding a new post.

At the end of their contract, postdocs have a 15% chance of being promoted into a permanent position.  That may sound harsh, but that’s actually slightly more generous than reality (the figures I’ve seen have it pegged at 12%).  Research track record doesn’t count in this scenario — this is a world where promotions are entirely a lucky coincidence (some would argue that this is broadly reflective of reality).  Once promoted, they’re now permanent academics and can apply for grants.

So here’s a sample run of the latest formulation of this scenario:

r_mean_pdr

Much like the last set of early results, we see a drastic drop in mean research output amongst permanent academics who are grant-holders, and postdocs don’t do very well in terms of productivity despite allocating 100% of their time to research.  Overall we see no benefit to research output of the population with the introduction of postdocs, and both permanent academics and postdocs see significant variability in their research output.  My interpretation is that the introduction of a randomised population of insecure researchers is massively disruptive — each semester we don’t know how good our postdocs will be, so their output is highly variable, and we also don’t know how good our promoted academics will be, so again we see fluctuations at that level too.

Scenario 4: Postdocs, Non-Random Promotions

This scenario is particularly intriguing to me.  Nic and I had wondered whether selecting only the very best postdocs from the crop for promotion each semester would improve the picture or not.  After all if we pick the best of the best and put them in a position to get grants and thus that juicy grant-holder output bonus, surely things will go much better for our virtual scientists?

Well… not massively:

r_mean_pdr

Now you’ll see in this run that actually both the grant-holders and postdocs appear to be doing a bit better in terms of research output.  Initially this seems good, but by the end of the simulation we see that the mean research productivity for the overall population is actually slightly lower than in the random promotions case!

At first blush this seems nonsensical, but if we ponder it for a moment I think it makes sense.  While the non-random promotions do mean that we get the best of the postdoc population promoted each semester, it still means we’re highly dependent on the whims of the random-number generator — if we get a few bad crops of postdocs, in other words, we just end up with more crappy academics, and our exacting knowledge of postdoc research quality hasn’t saved us from the disruptive influence of the constant influx of new people with highly variable research output and contract lengths.

Moreover, there’s no mechanism at present for postdocs to be mentored or to mature in their research abilities — once crappy, always crappy, in other words.  In real life people may argue that the trials and tribulations of postdoc life can allow young researchers to grow into more productive academics — so that’s another aspect we need to examine.

I’ve done a bunch more runs with different random seeds and seen variations in the output that seem to support these ideas, but I’m going to spare you the 18 other graphs.  Suffice to say that the graph above seems to indicate a lucky series of postdoc recruitment drives more than anything else.  Instead I’ll keep working at it and post more when I’m more clear on my interpretations of this scenario.

SURPRISE NEW SCENARIO: Non-Random Postdoc Promotions, With Mentoring Bonus!

Wow, what a day for you lucky people!  I’ve just decided to do a quick-and-dirty scenario where we give promoted postdocs in the non-random scenario a bonus to their research quality to attempt to simulate postdocs being mentored toward success by their superiors.  Surely we’ll see a change in the fortunes of our virtual scientists now?

Well… not really:

r_mean_pdr

In fact things look almost identical, with the exception of the overall mean research output hitting a plateau rather than dropping slightly at toward the end of the sim, as we saw above.

To be fair, however, the ‘mentoring bonus’ I gave out here was not outrageously large — effectively the promoted, mentored postdocs get a 25% bonus to research quality.  What if I double that to 50%, what do we get?

r_mean_pdr

Ah-ha!  At last, a very slight positive outcome.  Mean research output overall trends ever so slightly upward over the course of this simulation run, rather than plateauing or starting to fall as above.

But I think we’d have to admit here that this is a fairly minimal outcome considering a rather generous scenario — and it’s quite likely this won’t hold in every run and some other runs may show worse results depending on the feelings of the random-number generator.  It seems reasonably consistent at a quick glance — out of 10 runs I’ve just done for this scenario, 7 out of the 10 showed a similar tiny, tiny positive trend.

So what I’ve gathered from today’s work is that increasing the average research output of academics in a postdoc scenario requires some major work: we need to recruit only the very best postdocs; and we need to ensure they get mentoring of high enough quality that they are a full 50% better than they were during their postdoc days.  Even with these powerful tools, that’s still barely enough to overcome the disruptive impact of a fluctuating population of insecure overstressed young researchers.

In real life of course, we don’t have such a transparent method of evaluating research outputs and determining the best postdocs to hire — nor do we have a population of super-mentors who can massively improve the productivity of every single postdoc.  So, if we believe the underlying assumptions of this model, then perhaps we should start to think about whether insecure research posts are a good thing for science or not.

Of course there’s a human dimension here as well — over the many runs I’ve done with the postdoc mechanism running, most simulations top out around 500 active academics at the end of the simulation, and between 5-600 total postdocs hired over the 100 semesters.  Out of those we’ll see between 70-90 postdocs get promoted, while the rest all get the sack and leave academia forever.  Do we really want to be sending these vast numbers of PhD graduates out of the academy and lose all that potential research talent?  That seems like an incredible waste, and even more so when we see how difficult it is to get a positive impact on productivity out of this structure.

Next time: I’ll keep poking at this simulation and see whether these results hold up, and I’ll be doing some other comparisons on other measures, including total research output across different groups.  Early indicators: postdocs increase total research output, and research quality across the population becomes highly unstable.  More later.

 

 

Tagged , , , , ,