ParticipantCode,CodedBy,Extract,Code,AdditionalCodeInfo B1,je,"We’re up to 7 and 8s. They have prototypes, systems being demonstrated in the operational environment",TRL,7 B2,je,"I looked at it and I’d say that we’re between a TRL 7 and a 9 so on the chatbot NLP for users engaging with our platform to obtain a catalogue of items that they want to seek recommendations on distributed messaging applications with friends their conversations and recommendations, say we are TRL 9 with basic classification engine connecting to online stores including StoreName1 , StoreName2, StoreName3, StoreName4, I’d say we’re between a TRL 6 and 7 where it comes to the evolution of applying a graph based neural net native to our interest graph model and using a graph learning based approach to make a determination on how to almost dynamically or automatically evolve the display of catalogue items in an active conversation between users. So that’s kind of the evolution of where we are going. Our version 1 is out there and it’s in use. Version 2 would be an evolution TRL 6.",TRL,9 B4,je,some of this is happening inside organizations but its not happening at scale.,TRL,7 B4,je,"this document then outlines a number of case studies that are in that so like UniversityName have an algorithm that can do some coding, there’s a couple of generic ones that you’re familiar with there’s other companies like CompanyName is one of them, I can’t remember which one is which, that can do detection of melanoma off your SmartPhoneBrandName, err of your smart phone, and they have a number of clinical studies that have proven they are better at triaging. I think the question that they all have, and we’ve seen this for the multinationals, is right we’ve proven our technology in this area. How do we then operationalise that at scale?",TRL,7 B4,je,The hardest thing is getting it out of a research environment and deploying it in an operational setting,TRL,6 B5,je,"For sure, we have absolutely used it. So, in fact its been taken up… so one of the interesting things because of the err pandemic and the crisis that we’re facing right now, the technology actually can be used within the crisis.",TRL,9 B5,je,and attendance solution that is within 19 of the 32 or so councils,TRL,9 B5,je,rolling out this solution into the market,TRL,9 B6,vu,"Chat bot, prediction algorithms would be nice [V neural nets?] so the hyperparameter tuning is in 9 I would say as well. Weerrr, maybe that’s an 8. It’s in its final form, it meets its spec, its ready to go. We think it’ll work. It’s slow at the moment. So it does need a bit … So yeah, ours are in the bracket between 7 and 9, mostly 9",TRL,9 B6,vu,"when we left UniversityName we were .. some aspects that we had were 6 and some were 4. Like some were really like bets. Certainly stuff around, it actually does mention scalability there, but like whether you could do it anywhere else was an absolute gamble",TRL,6 D1,je,"Look at what a process looks like today, brainstorm how you might improve that current process",TRL,2 D1,je,a lot of our clients started to do AI projects on their own. They can do the technical proof of concept,TRL,3 D1,je,I think that stuff about creating a prototype and proving the value is a lot of the stuff that I do in the OrganizationalUnit,TRL,4 D1,je,"If we try to go into it on a technical challenge, we can do that sometimes on the experimentation side. They’re the ones that tend to be a technology project that then dies because you don’t get the adoption by the business because they go that’s very interesting that IT are predicting something but how am I going to use it?",TRL,4 D1,je,they kind of come to us to see is this something that’s even feasible,TRL,5 D1,je,"TRL 1 or 2, and then I guess with the lab we would follow them through to 4&5 probably",TRL,5 D1,je,"Then we do it on site with them and with real data. Typically, what we’ll do is we’ll do it in parallel to production.",TRL,5 D1,je,building those MVPs,TRL,6 D1,je,"So there its delivering the prototype so the success there is going to be on the client side, can see the product, can see the value and they’re getting excited about something that’s good to implement and there might be a bit of change management on that but when we get from prototype into early, alpha release beta release",TRL,6 D1,je,"To make sure we get that speed we need to do, we generally do something in parallel so we’ll export and import data or we’ll run it in parallel to production to prove it in the MVP. And then once its proven and we learn from that then we then say how do we productionize that and integrate it into production systems and we’re just starting to do that now on a few projects but we’re not quite there yet.",TRL,9 D1,je,"So, they provide you a service, you use it. Web APIs are the most common one where you’ll fire off on the internet to some request, so vision is a very common one at the moment, and they return you some information. There are also, you can just get models, you can buy models, pre-trained models. That’s another one. Anywhere you don’t, where someone is selling you something where you don’t need to input your data, is kind of what I consider to be off the shelf AI.",TRL,9 D2,vu,"The ones that get it, they go into the execution, so from probably TRL 5 through to that prototype, get the business buy in",TRL,7 D3,je,"TRL 4. There’s kind of only been the one project that’s advanced past that in the AI space, but yeah, most are TRL 4 [Interviewer: why?] I think a lot of the projects I’ve been working on have been kind of outside of an um real world environment. So its been trying to emulate real world conditions but within a kind of safe lab space, where there’s been kind of consideration towards designing thing to scale and work in the real world, but not necessarily having that completely proven",TRL,4 D3,je,trying to advance past the TRL 4 stage for an AI project,TRL,5 D3,je,"we kind of start with the ethnographic research. That’ll turn into a series of low fidelity prototypes that we will then evaluate with those users to kind of identify a candidate. And from that point we’ll try to build it out in high fidelity and check in on a fairly frequent basis. It changes per project. Typically, every kind of couple of weeks with the stakeholders to ensure that the direction of travel is still satisfying and solving the problem that those users are facing and then towards the end of that we um we try and get things in the hands of users so if we’ve built a kind of high fidelity, implemented prototype, we’ll try and give that to the end user and collect some feedback. And then typically that will be packaged up and delivered to the more kind of business stakeholder of the client",TRL,7 D4,vu,"Those which were, I would say those projects where the focus was on integration and management of internal data rather than on analysis, machine learning and all the other, well recently fashionable topics, some of them reached I would say TRL level nine",TRL,9 D4,vu,"for what is now considered AI, meaning machine learning and NLP and such things, those were, well, more like prototyping in first trial, so I would say levels between three and six seven",TRL,7 R1,vu,"we were improving the performance of their existing system, we were certainly at TRL 6",TRL,6 R2,vu,"Relatively AI savvy and using us to explore an opportunity, essentially. And I probably put that... That was almost just research really. I’d probably say that fits in the 2 or 3 area, I guess",TRL,3 R2,vu,"The company I spoke of which was doing satellite imagery analysis, so I would say we were more on 3 or possibly into 5",TRL,5 R2,vu,"At the other end, at the extreme, there are companies that say, look this is what I’m doing, and I know quite a lot about AI, I’ve got all this data and I can use it to improve my prediction of sales or to optimise the way I’m using my vehicles",TRL,9 R2,vu,"What are the sorts of things you want to do, where is your business going? Then maybe what kind of data do you have? What questions could you or do you want to ask on it? ",TRL,2 R3,je,a very simple algorithm that worked exceptionally well and that’s what we presented to them but it was just a feasibility study,TRL,4 R3,je,We produced a prototype which we demoed to them,TRL,6 R3,je,That was the one where we demoed. I forget where it was. Well we had the real data. I think its this one because we had it and we demoed it on the real data but not in their environment ,TRL,6 R4,vu,"a conceptual prototype perhaps a very basic software prototype. At which point the project ends. At that point on the companies are meant to fund further work themselves. So, let’s say from a TRL level this is erm kind of 4 5 6. It goes up to a demonstrator level let’s say",TRL,6 R4,vu,"enablers. So, these are pieces of software that SMEs should be able to take and should be at least at TRL level 6",TRL,6 R4,vu,"We take for granted that you have these components, these enablers already at a TRL level of 5 or 6",TRL,6 R4,vu,"with a bit of seed funding from us, push the TRL level up to say TRL level 8 or something like that. Sort of almost close to operational use",TRL,8 R4,vu,"the 8&9 bit was where we had done all the machine learning, there was the models there. What they were used to do was to integrate the models with their existing software.",TRL,9 R4,vu,"Obviously, this varies enormously so if you go to a company like WebTravelAgent that everybody uses, that its central offices are in CapitalCityName1 they have very sophisticated data analytics systems behind them",TRL,5 B3,vu,Redacted,CMM,3 B3,vu,Redacted,CMM,3 B3,vu,Redacted,CMM,3 D1,je,how do we de-risk it? It’s more of a phased approach,CMM,2 D1,je,"Part of our motto tends to be we want to be the client’s trusted digital transformation partner. So we’re not saying AI’s the answer what’s the question. We want to be in a position where clients come and speak to us, they tell us their business problem their challenges, then across our expertise we can say Here’s the right part of CompanyName to help with that, or here are two alternative solutions. And therefore you can say well is AI right or is analytics.",CMM,4 D2,vu,"when you get to that 10% that actually goes through to production success is usually pretty high, because if you’ve set those expectations right",CMM,2 D3,je,"we kind of start with the ethnographic research. That’ll turn into a series of low fidelity prototypes that we will then evaluate with those users to kind of identify a candidate. And from that point we’ll try to build it out in high fidelity and check in on a fairly frequent basis. It changes per project. Typically, every kind of couple of weeks with the stakeholders to ensure that the direction of travel is still satisfying and solving the problem that those users are facing and then towards the end of that we um we try and get things in the hands of users so if we’ve built a kind of high fidelity, implemented prototype, we’ll try and give that to the end user and collect some feedback. And then typically that will be packaged up and delivered to the more kind of business stakeholder of the client",CMM,3 R1,vu,"So they’ve formed a really rich partnership with us to do that. On that we are doing a number of different projects, and the idea will often be that we have an MSc student doing a project, and it goes well. They like the person and they’ll then hire them. That’s their method. But they again have a very clear vision that data science is key to the company’s future, and they’ve built a team internally that is sufficient to support that capacity.",CMM,2 R1,vu,that’s one where I would be concerned they would not be able to keep supporting it without further help to be honest. I left colleagues in charge of running that project at UniName. but I don’t know exactly what’s happened with that.,CMM,1 B1,je,you needed to have somebody sit there and tell you where the cat was in the picture. So we had a ground truth for each item of data.,People, B1,je,"There’s a paper I was looking at where they’ve done it for wine tasting. So I said again, OK what here is your measure of success. So again, it’s the Turing test. We’re going to have seven wine tasters and an artificial intelligent taster. Alright, so how’s that going to work then? Well, we’re going to, first of all we’re going to train, one of the panel are all going to train this AI taster that when it tastes this type of red wine, it will first of all be able to identify what the red wine is and then say whether this is a good version of that red wine or not, dependent on the chemical constituents and all this kind of stuff and then recommend what menu choices you have with it. OK. And then we’’ put that in with another group of wine tasters and they’ll say, everyone will then agree whether its 91 out of a hundred for this type of wine or whatever, and then we’ll have the wine taster, our artificial intelligence wine taster give a score as well and then recommend, make a menu recommendation as well. Apparently, it worked, indistinguishable from the other wine tasters. Or gave a value that’s within the bounds of what you would have expected. Its not completely mad, it’s identifying the right wine and was giving it an appropriate score and was able to make a menu recommendation. They went right that’s our success, a comparison with a human oracle.",People, B1,je,"there aren’t enough human oracles. Or there aren’t enough human oracles who are able to make a quantitative value that you can compare what an artificial intelligent value would be, because humans express themselves with this fuzzy language set.",People, B1,je,"in places where you do have an oracle it’s working. But that’s because I think you’ve got an oracle. So, the identification of breast cancer cells, I understand the current performance is as good as the radiographer in terms of the number of false positives and false negatives. But you had to have somebody who was a very good radiographer in the first place to train the AI. You needed that oracle to be in place to continually check and assess [interruption] … Yes so, the existence of an oracle or not and I would trust the radiographer, because they’re in a position of trust and they have a suitable qualified and experienced measure about them as to how or why they are an oracle, but would I trust another person on the street as a driver?",People, B1,je,"There’s a paper I was looking at where they’ve done it for wine tasting. So I said again, OK what here is your measure of success. So again, it’s the Turing test. We’re going to have seven wine tasters and an artificial intelligent taster. Alright, so how’s that going to work then? Well, we’re going to, first of all we’re going to train, one of the panel are all going to train this AI taster that when it tastes this type of red wine, it will first of all be able to identify what the red wine is and then say whether this is a good version of that red wine or not, dependent on the chemical constituents and all this kind of stuff and then recommend what menu choices you have with it. OK. And then we’’ put that in with another group of wine tasters and they’ll say, everyone will then agree whether its 91 out of a hundred for this type of wine or whatever, and then we’ll have the wine taster, our artificial intelligence wine taster give a score as well and then recommend, make a menu recommendation as well. Apparently, it worked, indistinguishable from the other wine tasters. Or gave a value that’s within the bounds of what you would have expected. Its not completely mad, it’s identifying the right wine and was giving it an appropriate score and was able to make a menu recommendation. They went right that’s our success, a comparison with a human oracle.",People, B1,je,"in places where you do have an oracle it’s working. But that’s because I think you’ve got an oracle. So, the identification of breast cancer cells, I understand the current performance is as good as the radiographer in terms of the number of false positives and false negatives. But you had to have somebody who was a very good radiographer in the first place to train the AI. You needed that oracle to be in place to continually check and assess [interruption] … Yes so, the existence of an oracle or not and I would trust the radiographer, because they’re in a position of trust and they have a suitable qualified and experienced measure about them as to how or why they are an oracle, but would I trust another person on the street as a driver?",People, B2,je,"how do you determine what will be serendipitous to a user, what will be relevant to a user",People, B2,je,"we did engage an external consultant to help us construct it initially and then since that initial work we’ve had our engineers train and set up an in house online certification and training house so that we’ve been able to work through the iterations and latest releases, and models and constructing that. So its, we have a pretty young team. We bring in new engineers fresh out of undergrad, especially every year",People, B3,vu,Redacted,People, B3,vu,Redacted,People, B3,vu,Redacted,People, B4,je,"all the chief executives, all the regulatory bodies, the first time I think in years they’ve all sat together in the same room,",People, B4,je,"How do we then create the kind of SWOT teams who will support implementation. So, do we have the data scientists inside these organizations? Research, I can send you some of the papers, shows that we are lacking in informaticians never mind data scientists, in side our national health system",People, B4,je,"a lot of clinicians will say that the biggest problem they have is even if they are interested in adoption of AI technology in cardiology, for example, it’s difficult for them to get backfill inside their organizations to free up their own time to continue further work outside of... They can’t do it in their day job",People, B4,je,"When you have a workforce that struggle to use digital technology appropriately inside their organizations, who are doing healthcare improvements [unclear – science?] and struggle to understand video conferencing or analytics its very difficult for them to think through “in my improvement world, inside a local organization, how might I think through “there’s a better way of doing this other than the way I’ve currently been doing it, with the use of technology””",People, B4,je,"we need a workforce that understands what’s capable today, with that future focus goal",People, B5,je,"is down to the organization, and if they don’t put good and smart people onto this who understand how it’s supposed to work and engage with their customers, right?",People, B5,je,"a CountryName1 perspective, and the ethical requirement are a little different between CountryName1 and CountryName2 and that time, and because we weren’t going for the HealthOrganisationName we didn’t need to use the HealthOrganisatioName ethical approval process, which is entirely cumbersome. What we did what we did was we used the ethical processes of the individual councils that we were working with at the time. So the councils had to ethically approve it. Now fortunately we were working with some very smart, initially, some very smart people in the first council and they had their own ethical err process and a specific person that was managing that process, so we worked very closely with her to get that first ethical approval over the line. And so from there what we did was we identified the process we had to go through, and we did that. But on the data protection perspective, we make sure that everything is anonymised at source.",People, B5,je,"protection, the European data protection policy as a basic erm [pause] barrier to any kind of collection of data. Its not a barrier. You can do whatever you like if they fully understand what you’re doing and agree with it.",People, B5,je,putting my metrics in with a known outcome and that known outcome was given to it by me because I know what good looks like and I know what bad looks like.,People, B6,vu,"when I said about software engineering, we were very I think fortunate, and careful, that the first few people that we hired on the data science end had solid background in deployment and … we were looking for testing but we didn’t really get it and we definitely got like exBigITCompanyName2 graduates that could wrap their head around some of the more advanced bits and bobs that they needed",People, B6,vu,OK so the big one is outages. Someone’s turned something off. Arrr and that happens often. You know that just happens cos it’s the real world and yeah. I know somebody who works in the space industry and like they don’t have people hoovering in space and they go and unplug your kit every Tuesday so like [laughs] that’s a common one.,People, B6,vu,anomalies that the energy manager should go and worry about as opposed to just quirks,People, B6,vu,"uniquely for most businesses, so far, and I think this is unsustainable for a scale out proposition we talk to nearly every level of our customers’ organizations.",People, B6,vu,"I’ve really emphasised on them it being a really crucial part of our business because they hold the knowledge that the AI will never have. Like they know, so let’s pick an example of that. So we got a site in CityName. It’s got a natural ventilation thing in the roof. So when its not raining and it hot it’ll open the window in the roof, but they override it sometimes and that means it overheats. So it looks... so the machine learning goes ‘waorrs, you’ve messed up there guys! You’ve done a bad.’ But it’s because there’s a cotton tree outside and the cotton blossom blows in. That guy that knows about the cotton tree blossom, he’s the thing that stops you looking like an idiot, right? When you’re the machine learning guy. ",People, B6,vu,"Like that’s the whole point of the company – it’s IP. I really don’t. That’s, that’s a lie! And eh um, I think that it’ll mature. I think that as you get more informed people",People, B6,vu,it moves that individual out of being overheads into being able to talk about the core business.,People, B6,vu,being a really crucial part of our business because they hold the knowledge that the AI will never have,People, D1,je,over to more robust delivery team,People, D1,je,"we’re talking to the CIO in the IT department they just want to do an IT project, but if it’s a business user they’re the ones who get the real benefit",People, D1,je,"we’re not implementing AI visibly, we’re hiding it, but behind like a better service, and that’s then more palatable for the end users.",People, D1,je,So its getting stakeholders from both senior and the ground level people who’ll be using any kind of solution.,People, D1,je,"we had an A&E consultant, a discharge nurse, we had IT in the room",People, D1,je,"if we can get the exec sponsor to say no jobs are going to be lost by this, its basically to help us to support growth, that really helps the job security and the internal fightings",People, D1,je,have we got a senior person that’s going to take this through,People, D1,je,a head of development,People, D1,je,"We’ve seen a big growth in the last couple of years in terms of training more people in data science and AI. [unclear] investment there. But also the interest in learning that kind of side of things, especially from the kind of younger people in the company it has really gone through the roof. Every kind of graduate I talk to has an interest in doing work with AI and doing work with data science. And its just because its new fun flashy technology but there is a real interest I’ve seen which is great. So we actually have quite a lot of skill in the company now than we did a year and a half ago. And I think we have quite a lot of skills in our company compared to a lot of the people we have come in don’t necessarily have trained people in AI specifically. They will often have maybe some data scientists or analytics team, but there’s not a lot of training in specifically AI and the more advanced technologies",People, D3,je,clients trying to use AI and emerging technologies to help their employees focus on the more challenging more interesting parts of the work and automate some of the necessities. So we’ve seen clients in the retail space especially looking to try and take away the repetitive parts of the job that machine learning and AI can learn to do and enable those employees to spend more time with customers,People, D3,je,enable improved experiences for employees,People, D3,je,"we’ve had a kind of blended team. So we’ve brought in domain experts from the client, so no technical experts but they completely understand the problem, and then we’ve supplemented that with our kind of user experience researchers and our technologists. And now we’re starting to bring in technologists from the client side as well so we can transfer the knowledge as we’re progressing",People, D3,je,"starting to train their IT staff and the broader…, all of the kind of stakeholders within that system. So the people who actually have to interact with the data, through to those who kind of manage those processes. So in the past we’ve tried to bring those people along and educate them as to the reasons that we are trying to recommend these more open data approaches",People, D3,je,"ethnographic research. So we spend a lot of time with the individuals we’re trying to solve for. So we always ensure there’s a real problem and we’re not applying   technology for the sake of technology. Because we’ve found that that doesn’t tend to progress particularly quickly. So we always try to bring in the end users right from the start, understand their day to day, understand their key pain points, and try to work out how we integrate emerging technology or AI into that to try to solve those problems in the kind of least intrusive way that we can, so we’ll kind of create low fidelity prototypes and then test those with the people who actually need to use them to see which fits into their kind of work schedule",People, D3,je,we’d bring in our experts from our side in that kind of field to help them improve their data management before then taking the next step and applying machine learning,People, D4,vu,If the project was limited to the knowledge and data management it was just enough to have maybe a couple of people who knew what the scene was about and were able to work with external contractors,People, D4,vu,"In case of really ambitious projects where they wanted to not just integrate data but had also been able to exploit it for machine learning, inferencing and so on, yeah, as well as integrating this err, new information into their … databases, in that case organizations usually tried to also bring in, well people with appropriate skills into their internal workforce",People, D4,vu,"people working internally and people to understand what the technologies are, willing to lobby for them in front of their higher management and err, well, just thinking, probably [pause]. Yeah, it required a few people with knowledge, enthusiasm, and enthusiasm for technology in management positions.",People, D4,vu,"Well having sufficiently enthusiastic people within the organization who are able to promote it, and being able to select the really good use case … with which it would be able to impress the, a few target users, target ends users, who err, who did the first try of the results",People, R1,vu,"They were all mechanical engineers in the project. But what they didn’t have was experience of data mining so they were quite technically literate and knew what they needed, but didn’t have the skills to fill that gap.",People, R1,vu,"the CEO has decided, yes we need to be, data science is the key to our future",People, R1,VU,"So they’ve formed a really rich partnership with us to do that. On that we are doing a number of different projects, and the idea will often be that we have an MSc student doing a project, and it goes well. They like the person and they’ll then hire them. That’s their method. But they again have a very clear vision that data science is key to the company’s future, and they’ve built a team internally that is sufficient to support that capacity",People, R3,je,"IT, head of IT whatever the title was within the company. Head of IT and the second person was his deputy.",People, R3,je,"we had this more regular contact with the IT people and then when we got to visit we of course got to see, and we had a postdoc who did lots of interviews with stakeholders at every level in the organization",People, R3,je,"they hired a person with a masters in data science to do it for them. And I don’t remember for how long they said they have had that person on board and they couldn’t do the job. And that’s when they realised that this is actually such a big problem, and complex problem, that somebody that’s got a relatively basic training skills will not be able to solve.",People, R3,je,"we had the CEO, no the chief technical officer, in all the important meetings, the head of IT, and the person who’s going to be the day to day manager. We went to speak to people in the call centre, so that’s even before writing the proposal, they were so open about what they had",People, R4,vu,"you have a strange situation where small companies where everybody knows things typically don’t have the resources to do much research or investigation. Big companies that do have the resources have a very poor ability to communicate with each other internally, so the left hand might be doing something quite innovative and the right hand doesn’t have a clue about what’s going on.",People, R4,vu,"People were in a state of panic that they didn’t know anything about buzzwords – ‘we’d better sit in here, we don’t have a clue, we don’t understand anything, yes let’s participate, let’s be involved’.",People, R4,vu,"People who know about the real world and how the real world operate don’t really know very much about technology, OK. People who know about technology, who are typically in their late teens or twenties, may have done a technical degree don’t really know very much about the way the real world operates right? So, you’ve got like these two communities that by-pass each other",People, B1,je,"there’s a legal obligation to do safety arguments to justify why something is safe to use. And there are health & safety at work act which says you have to do a sufficient risk assessment. So anything that gets used by anybody in a workplace has to have had a sufficient risk assessment done on it. The that has to be recorded and written up, in what’s called a safety case, and a case as in a legal case – a legal case for its safety. So that when the employer gives it to the employee, the employee is not going to be harmed by the equipment they need to do their job. So there is a requirement to do this risk assessment and in some high risk industries it’s been specified that you have to do a safety case. So high risk industry would be nuclear, or oil and gas and, as it happen the military, with defence because of the use of energetic weapons and kinetic effects. But also building sites, lifting equipment., things that can kill people",Process,b B1,je,"Alright if it’s an AI coffee machine or something like that we probably don’t need to go very hard over on the safety argument for it, but if this is now going to be a heavy vehicle that’s moving, it’s got an energy, if it’s a high energy industry, governed by artificial intelligence or autonomous behaviour, machine learnt behaviour, we’re going to have to satisfy the legal requirement for a new set of safety arguments.",Process,b B1,je,"But if, the theoretical bit has been given to me but we want to have 50 autonomous vehicles on the ground supported by the 150 autonomous drones, autonomous air vehicles or AI based vehicles and what we want to do is try and capture that flag. We’re going to rehearse this as an operational concept and in fact they’ve said, so the operational defence one is, so we want to make that flag surrender.",Process,b B1,je,"The civilian one is we’ve got to get all these supplies to that cut off village and so we want 50 vehicles on the land who can each take a load of 100kg each of food or medical supplies, and they can then convert into amphibious ones as its 2 miles of flood to get to that village and then we want to medivac out two people that have got hypothermia, or some other, whatever.",Process,b B1,je,"one of your success criteria for a human is can you pass the driving test. You’re tested for one hour, in your home town, or nearby to your home town, with an oracle by the side of you that will look at how you behave on the road and that kind of thing. And they’ll say, yeah you’re passed and you can go off and drive on your own then. So if I’ve got my AI CompanyName car, or whatever through a driving test, and it passes the driving test are you happy for it to go on the road now?",Process,d B1,je,"There’s a paper I was looking at where they’ve done it for wine tasting. So I said again, OK what here is your measure of success. So again, it’s the Turing test. We’re going to have seven wine tasters and an artificial intelligent taster. Alright, so how’s that going to work then? Well, we’re going to, first of all we’re going to train, one of the panel are all going to train this AI taster that when it tastes this type of red wine, it will first of all be able to identify what the red wine is and then say whether this is a good version of that red wine or not, dependent on the chemical constituents and all this kind of stuff and then recommend what menu choices you have with it. OK. And then we’’ put that in with another group of wine tasters and they’ll say, everyone will then agree whether its 91 out of a hundred for this type of wine or whatever, and then we’ll have the wine taster, our artificial intelligence wine taster give a score as well and then recommend, make a menu recommendation as well. Apparently, it worked, indistinguishable from the other wine tasters. Or gave a value that’s within the bounds of what you would have expected. Its not completely mad, it’s identifying the right wine and was giving it an appropriate score and was able to make a menu recommendation. They went right that’s our success, a comparison with a human oracle.",Process,d B1,je,"in places where you do have an oracle it’s working. But that’s because I think you’ve got an oracle. So, the identification of breast cancer cells, I understand the current performance is as good as the radiographer in terms of the number of false positives and false negatives. But you had to have somebody who was a very good radiographer in the first place to train the AI. You needed that oracle to be in place to continually check and assess [interruption] … Yes so, the existence of an oracle or not and I would trust the radiographer, because they’re in a position of trust and they have a suitable qualified and experienced measure about them as to how or why they are an oracle, but would I trust another person on the street as a driver?",Process,d B2,je,"they explicitly enable our app to have access to their product catalogue to give them the working features, capabilities that they need.",Process,b B2,je,"we have a mobile telco customer who needed additional end user customer care management in one of their messaging, large volume messaging applications. So we applied a sentiment analysis model that could understand the sentiment of users that were interacting with the service platform, so that it could mediate when user had unhappy experiences with the services they were subscribed to or consuming, were trying to, were expressing their reactions, as it were, to the system expecting that a human being would read it and react where there was no human there",Process,b B2,je,"from user to user or peer based messaging applications that interface with the graph for dynamically generating AI created catalogues which match the subject of an active conversation users are in and basically models keywords into attributes and phrases which are useful to rendering an update of the catalogue that users are actively having a conversation about, and giving them more options to make recommendations to each other on.",Process,b&d B2,je,"We certainly have refined our approach to learning on a project or a needs basis so when a specific project or platform, product platform development we’ve found it necessary to change our approach to how we construct, acquire learning and so on, we have .. so it’s been more on a needs basis than a general reorientation, how we learn. We generally persist a constant learning model and its just really hard to stay relevant in a highly, in a fast changing technology market, if you’re not constantly learning on every level.",Process,d B2,je,"we do all the data curation automatically by figuring out where the knowledge base items and the product catalogue items fit in our existing data graph model. So our data graph platform basically defines what the node edge relationship structures are and so when we ingest a particular customer model it has an interface for determining where the say product catalogue configuration for an online retail brands fits, matches, and is represented within our existing graph model. So it’s kind of an automated graph learning approach where our existing model has an understanding of lots of genres in the existing domain and so its kind of matching recognition between the dataset of a new online brand activating on our application.",Process,d B2,je,"In some cases, where they have lots of existing applications and data warehousing and all of that. Where we have to actually specifically study what they have and then construct a unique model, again to give them value, even where we are constructing a unique model we interface it to our existing data graph model, just because of how expansive it is in understanding across different domains and different data structures. So we ultimately have been involving an automated process for answering the question, where does this new data sit? If it’s in our existing graph, and so it’s basically graph learning prediction model, where every new node you’re bringing in you try and establish what it should have an edge connection to.",Process,d B2,je,"In some cases, where they have lots of existing applications and data warehousing and all of that. Where we have to actually specifically study what they have and then construct a unique model, again to give them value, even where we are constructing a unique model we interface it to our existing data graph model, just because of how expansive it is in understanding across different domains and different data structures. So we ultimately have been involving an automated process for answering the question, where does this new data sit? If it’s in our existing graph, and so it’s basically graph learning prediction model, where every new node you’re bringing in you try and establish what it should have an edge connection to.",Process,d B3,vu,Redacted,Process,b B3,vu,Redacted,Process,b&d B3,vu,Redacted,Process,d B5,je,their home and at least you can then see their activity levels and whether they’re declining and whether they are able to cope,Process,b B5,je,"is down to the organization, and if they don’t put good and smart people onto this who understand how it’s supposed to work and engage with their customers, right?",Process,b B5,je,"stratify them to high risk, medium risk and low risk.",Process,b B5,je,"is output to the patients as a management function not as a life critical system,",Process,b B5,je,"a device that’s completed by the pharmacist, and its disposable apart from the electronics that sit on the top and control the thing. It connects through GPS and effectively its basically a timer and the paper or cardboard disposable unit has rows and rows of spaces for your pills with morning afternoon evening and night time, and as you break the seal that then electrically registers that you’ve taken your pills. So its able to monitor and feedback compliance.",Process,b B5,je,"say, can we collect more data in the community and then start to create that algorithm from this around a machine learning tool so that we could automate that process",Process,b&d B5,je,So this is why we want to supply these flags and indicators. So that these flags and indicators highlight well before hospitalisation is necessary. Because what we’re trying to do is manage you in the community and mitigate the cost.,Process,b&d B5,je,putting my metrics in with a known outcome and that known outcome was given to it by me because I know what good looks like and I know what bad looks like.,Process,d B5,je,Pretty much every metric that I’ve used has a solid basis in research from at least one or two papers,Process,d B6,vu,it’s for what I call “speed to value”. So when a customer places an order how quickly can you produce a reliable prediction is quite a driver for commercial growth in the company.,Process,b B6,vu,a big part of the product is around what I call weird thing spotter. And that is about working out when you’ve left the heating on overnight basically is a good example,Process,b B6,vu,we do have a customer journey of course. There’s quite a lot of on boarding,Process,b B6,vu,anomalies that the energy manager should go and worry about as opposed to just quirks,Process,b B6,vu,"there is a moment, there is an “unboxing moment” for us, which I’m doing a bit of work on, about how you make it feel like... like an UpmarketITProductName right? It’s got to feel nice, right?",Process,b D2,vu,"the disrupter, where they’re trying to find new models of software and business using these techniques to disrupt the market. So to bring new capabilities, new apps, new insights takes those kind of things",Process,b D2,vu,"the optimisers. So they’re the people who would just go on from data science and just go optimise, optimise, optimise against current business goals. So the ones who are going can we get people to buy more using more targeted advertising, can we cut costs, can we reconfigure our supply chain, can we do predictive maintenance. It’s very much that optimisation mind set.",Process,b D2,vu,"If the data was in the right shape it means somebody’s put it in the right shape already, which means they’ve done a similar piece of work.",Process,d D3,je,using blockchain in the travel industry to try and decentralise your travel agent,Process,b D3,je,say machine learning and computer vision in the context of a retail store and to understand the activities that people undertake within the store and who those people are,Process,b D3,je,a lot of work to be done in how to roll out technology such as AI in a way that is less of a huge change for the kind of day to day process,Process,b&d D3,je,"ethnographic research. So we spend a lot of time with the individuals we’re trying to solve for. So we always ensure there’s a real problem and we’re not applying   technology for the sake of technology. Because we’ve found that that doesn’t tend to progress particularly quickly. So we always try to bring in the end users right from the start, understand their day to day, understand their key pain points, and try to work out how we integrate emerging technology or AI into that to try to solve those problems in the kind of least intrusive way that we can, so we’ll kind of create low fidelity prototypes and then test those with the people who actually need to use them to see which fits into their kind of work schedule",Process,d D3,je,we’d bring in our experts from our side in that kind of field to help them improve their data management before then taking the next step and applying machine learning,Process,d D3,je,"we kind of start with the ethnographic research. That’ll turn into a series of low fidelity prototypes that we will then evaluate with those users to kind of identify a candidate. And from that point we’ll try to build it out in high fidelity and check in on a fairly frequent basis. It changes per project. Typically, every kind of couple of weeks with the stakeholders to ensure that the direction of travel is still satisfying and solving the problem that those users are facing and then towards the end of that we um we try and get things in the hands of users so if we’ve built a kind of high fidelity, implemented prototype, we’ll try and give that to the end user and collect some feedback. And then typically that will be packaged up and delivered to the more kind of business stakeholder of the client",Process,d D3,je,"we’re trying to get maybe err smaller scoped projects into the real world slightly faster, to try to make little and often change rather than a kind of, a huge change.",Process, D4,vu,"Like, let’s try this to do this small piece, and see how it looks. Whether it works for us. Whether it brings us value, and potentially can bring us value, and then we decide internally whether we move to the next stage or not. So, yah, its, err, both internal objective, err, decision making process, as well as of course, the usual bureaucratic procedures in such big organizations",Process,d R1,vu,"The way that we do it is that we build the models, we do the innovation. We test out a variety of techniques and evaluate them. We then say this is what we think is the right model. This is how it should be retrained. They then integrate it into what they are doing",Process,d R2,vu,"Optimisation is a big thing, route planning",Process,b R2,vu,could they then make smart predictions about so they can do pre-emptive delivery or collection to someone,Process,b R2,vu,they wanted to monetize that and then be able to mine the data for interesting information about their demographic and then maybe be able to sell that to all sorts of people,Process,b R2,vu,route planning – where do I position my product to minimise the journey,Process,b R2,vu,a lot of companies can see the potential for the data they’re gathering to be used to advise someone for their marketing campaigns or their sales campaigns. And that kind of makes it visualisation as well,Process,b R2,vu,"having to do the boring planning and how you structure your data, where you put it",Process,d R3,je,a consortium of banks doing a feasibility study of trying to get some consensus around how to do fraud detection more effectively,Process,b R3,je,"operations management on the site as well as doing predictions for what they might need to do at the end of the day. Because of the way they operated Their signing up and [Interviewer: literally end of day?]. Yes, end of the working day or kind of like beginning of the transport day which was 9 o’clock in the evening, and try to do predictions at different times in the day because things come in",Process,b R3,je,the territories in the UK were distributed between them in the sort like there is a sharp line between this and this but then they could still take each other’s parcels if they were not too distant from each other and they had no control over who phones up which haulier company and if they had a border line it could have gone that way,Process,b R3,je,"a very, very complex scheduling problem with lots of constraints and things that change dynamically. So this is a company that is maintaining equipment, mostly [unclear] oriented but also equipment for big clients and they also do cleaning and other things, so services. But most of it is maintenance of electrical equipment, and they have engineers in the van going around in the country",Process,b R4,vu,"with the application of machine learning or deep learning what is it that you are doing and what is the digital data that you could use? Err... And then we can start saying well would you like to predict something, do you want to analyse something, do you want to optimise in some way some process, some business process? So these are, I would say one thing is understanding what your business activity is in terms of real world vs digital.",Process,b R4,vu,"actually understand your business process, your workflow in order to see whether actually there is any point in applying this err any particular version of AI",Process,b R4,vu,At which point somebody will look at the container and say Ah! That’s actually supposed to go to CapitalCityName3! [laughs] at which point it might get trans-shipped,Process,b R4,vu,"So, company foundation documents and financial reports. These of course are produced in many different languages across the ContinentName  but whenever you have cross border interactions, whether its mergers, whether it’s some kind of thing the classic scenario is you go to your lawyer in CountryName1. CountryName1 lawyer tries to find a lawyer in CountryName2 or CountryName3 or somewhere and says ‘I need the documents for this company’ and they will go to chamber of commerce or wherever these things are, get them, then they have to be translated and all this. So this process is quite complex and cumbersome, and this start-up wants to automate this process using machine translation and things like that. But what’s interesting is although they have the idea ‘we need to simplify this process to make it easier, cheaper, more efficient, and we can use AI’",Process,b R4,vu,"WebTranslationApp is based on hundreds of billions of pages of training data, right? You’ve got that problem yes? Have you got the training data? Have you got the mechanism to acquire the training data? Are you prepared to create a workflow, an interface, a platform, where people can, you know go through a series of steps, ask for specific services, and in the process also capture that data so you can subsequently improve your pro...[breaks off in middle of word].",Process,b&d B1,je,"we need like 50,000 channels of information, um. And we need that in every, well we need that at 10Hz type of thing, and then we need to store that. Where are we going to store that much information? How are we going to record that? The bandwidth to record that is enormous",Technology, B1,je,"So the pictures that were taken on this particular scanner type, this scanner was set up this particular day to get this level of resolution in this particular area of the brain and it has this type of colour match, but the countryName3 made machine has a different set of colours, and it has a slightly different size of screen because they use sixteen nine for their screens whereas CountryName use ten eight for their screens because theirs are based on CountryName paper sizes and RegionName ones are based on RegionName paper sizes.",Technology, B1,je,"And they’ve had the same trouble with the vehicles. So they’ve got these multiple sensors, they’ve got radar, the visual and what they call a lidar, yup, you know all about that. And what they’ve found is the lidar operates at 25Hz, but this bit operates at 20Hz, so somewhere on the lidar I’ve got 5 extra images here that I don’t have an equivalent one for. So when I’m trying to integrate those 20 images per second from that with the 25 images per second from this. [V all in the same vehicle?] Yes so you have 3 different sensors on it. And I’m trying to match up now. That timing band doesn’t work with that timing band. In fact it now looks like that image came 0.02 sec apart from that image, so is that tree still there or not?",Technology, B2,je,building an appropriate client to our graph based recommendations platform,Technology, B2,je," a chat bot in a messaging platform, initially in CommunicationAppication would be more appropriate as a channel",Technology, B2,je,"So we had to basically have a classification engine. So the classification work naturally you fold into constructing multilayer perceptrons, and that would be the second leg of our work involving an AI.",Technology, B2,je,"form user to user or peer based messaging applications that interface with the graph for dynamically generating AI created catalogues which match the subject of an active conversation users are in and basically models keywords into attributes and phrases which are useful to rendering an update of the catalogue that users are actively having a conversation about, and giving them more options to make recommendations to each other on",Technology, B2,je,"Everything from understanding graph theory and learning NLP and ML on different platforms from PlatformName to CodingLanguage-based frameworks and Acronym, CodeLibraryName. So it’s a constant learning environment. At the initial stage of constructing a graph database on the GraphDatabasePackageName.",Technology, B2,je,an intermediary sentiment analysis ML application,Technology, B2,je,classification engine related to our BrandName social recommendations graph platform,Technology, B2,je,an augmented reality interface that was embedded in the chatbot for social recommendations,Technology, B2,je,some data graph modelling that would ultimately be used in genre classification also for online retail,Technology, B2,je,they’ve implemented an API on the way their product catalogue model is arranged,Technology, B2,je,"maturing the graph learning model. So its err… which on the one hand involves enriching the graph data structure itself, and on the other hand being able to adopt and adapt the best ML components and libraries that can achieve the kind of multi-typed graph learning which we’re constructing",Technology, B2,je,an augmented reality interface that was embedded in the chatbot for social recommendations,Technology, B5,je,"they would give you a medical device costing two and a half thousand pounds that you would strap to yourself. It would monitor your heart rate with these… with the three electrodes and I was aware, and I had one in fact but I hadn’t thought of it, was a polar device. So what I was measuring was what we call the R to R interval, not the ECG but the R to R interval, which is the next level down. And what I noticed was, that was incredibly informative and I didn’t need an ECG to see my performance improvements. Soo.. and that was an off the shelf fifty pound device, chest wearable strap that I could monitor that using some software that was free off the internet, you know from the app store, and I could monitor my performance",Technology, B5,je,"spreadsheet and just number crunch it. Or create an algorithm and go stepwise through it and create flags and pointers. But actually having a model where you can create data. And so my data that I did, and I’m skipping forward a little bit, but the data that I used it was learning so I created a … a model that was learnt. So this wasn’t a case of, I knew what I wanted to do and the only, really, way to create it was to be, erm, almost through a quasi-expert system",Technology, B5,je,"gathered on I guess a commercial system, we use BrandName devices and so BrandName have an application on a phone that gathers the metrics directly from their wearable device and we use an API in the background to harvest the data from the BrandName system",Technology, B5,je,"a device that’s completed by the pharmacist, and its disposable apart from the electronics that sit on the top and control the thing. It connects through GPS and effectively its basically a timer and the paper or cardboard disposable unit has rows and rows of spaces for your pills with morning afternoon evening and night time, and as you break the seal that then electrically registers that you’ve taken your pills. So its able to monitor and feedback compliance.",Technology, B6,vu,can you make a prediction of what the power consumption profile of that building,Technology, B6,vu,"the temperature sensors that we take temperature readings from there were installed in 1998, which is, what’s that? Dot com bubble right? It’s before the dot com bubble burst, it’s before SocialMediaPlatformName, it’s before cybersecurity was a word. It’ a really, really long time ago and we have to interface with all of those devices so you have some reliability issues",Technology, B6,vu,"So what we actually find is our model works better using something called LearningAlgorithmName1, when we have less data. And as time passes and we’re getting more data we find that LearningAlgorithmName2 outperforms our LearningAlgorithmName1 algorithm. And we have considered hopping back occasionally to the LearningAlgorithmName1 model as well when the LearningAlgorithmName1 outperforms the LearningAlgorithmName2 .",Technology, B6,vu,"So there’s a chat bot on the product and that’s something that’s not built by us. We borrow BigITCompanyName’s. Buy a ProductName2 product. That’s a creepily, terrifyingly good bot",Technology, B6,vu,"Reliability, resilience and uptime",Technology, B6,vu,"I think success for the company is, yeah, its built around scale. It means, whatever flavour it kind of comes out in the end, and that’ the business is in the middle of that [laughs]. It means that this tech works on as many assets as we can get it to point to.",Technology, D2,vu,"people will push it back into software development, because that’s what always happens. Search, when I learnt AI, was part of AI. It’s now what a search engine does and that’s computer science. And so search is now computer science, it’s not AI anymore because we can do it. And that has happened with AI loads of times.",Technology, D3,je,designing and building the tech core solutions,Technology, R1,vu,this is what we think is the right model. This is how it should be retrained. ,Technology, R1,vu,they have a whole pipeline for creating models and putting them into practise,Technology, R2,vu,"The smart city, in terms of there’ll be instruments, cabinets they measure traffic and target advertising. That’s kind of a big thing",Technology, R2,vu,They developed a smart water meter I think. Then they were interested in how water meters and electricity meters and gas meters,Technology, R3,je,"towards the end of the project they purchased some equipment that allowed them to monitor what was happening much better, and that could have been part of the reason why they commissioned a solution, a commercial solution anyway, because that equipment allowed them to capture data better and they wanted to build that in",Technology, R3,je,"wasn’t quite AI but was still a very visual and technical solution. Sort of colour coded what they can expect where and traffic light system colour coding and yeah. [Interviewer: data analysis system?] Yes data, prediction, demand prediction system. a lot of different little things but in terms of AI it was basically a prediction system. So if you like a regression machine learning it was a regression problem",Technology, R4,vu,we’re doing AI but we’re calling it robotics.’,Technology, R4,vu,"Using GPS you can do that. But you’d like to do a planning of the route of that tractor using image processing or images that come from drones or satellites that allow you to determine what the growth is across the field, which parts of the field are growing successfully, which parts of the field have insufficient fertilizer or things like that. All that computing power is going to take up a lot of, you’re going to need a lot of computing power to do basically a travelling salesman problem solution, on the results of your image processing analysis. Now if you’re in a geographically remote area, you’re either going to have to have that computing power at the edge of the system, that is on the field or on the farm. Or you’re going to have to transmit these images and all that data up somewhere, and you can’t because you don’t have broadband in the remote area and you cannot possibly afford to send it through satellite transmission",Technology, R4,vu,"here in CountryName where we have 5G or a 5G rollout to do that sort of thing. But it’s not going to be possible in most areas of CountryName2 because nobody’s going to put that kind of data transmission capability at a local level. So, then you have to say how do I do computer processing of that level of complexity right locally on farm",Technology, B1,je,"there are lots of difficulties with having enough data that, so that you item, or machine learned technology um doesn’t frequently come up with out of distribution values",Data, B1,je,you needed to have somebody sit there and tell you where the cat was in the picture. So we had a ground truth for each item of data.,Data, B2,je,"In the case of a strategic customer, usually we go through a process of defining what datasets and profiles will be required to deliver the solution they want. And then they would normally go through whatever organizational policies and structures they have to not only approve our access but also share whatever access rules which they would require us to adhere to. And that’s usually a unique scenario. Sometimes it involves a case where we have access to the data by virtue of an application or service we have rolled out which has generated this data. And then we’re generally introducing an AI based application as a secondary layer or secondary value add to what the primary application is.",Data, B2,je,"we do all the data curation automatically by figuring out where the knowledge base items and the product catalogue items fit in our existing data graph model. So our data graph platform basically defines what the node edge relationship structures are and so when we ingest a particular customer model it has an interface for determining where the say product catalogue configuration for an online retail brands fits, matches, and is represented within our existing graph model. So it’s kind of an automated graph learning approach where our existing model has an understanding of lots of genres in the existing domain and so its kind of matching recognition between the dataset of a new online brand activating on our application.",Data, B2,je,"However, on data privacy and security basis we do have subgraph unique segments where we understand what is unique to a particular customer engaging with our graph platform",Data, B3,vu,Redacted,Data, B3,vu,Redacted,Data, B3,vu,Redacted,Data, B3,vu,Redacted,Data, B4,je,"We have issues with our data governance, we have data access, we have fragmentations of how things are coded [medical coding?], interoperability, you know how one system talks to another. These are all ongoing issues that happen across digital health technologies and they’re exasperated [means exacerbated?] when we start to look at AI",Data, B4,je,some of this is happening inside organizations but its not happening at scale.,Data, B4,je,"data quality is an issue, erm, and I use the word loosely but interoperability. So just because its put into one system doesn’t meant that’s going to be the same in another. When you’re training an algorithm you know it might work on platform A but never work on platform B unless you retrain it.",Data, B4,je,"You can’t turn around and say “give me your data dump! I want to do something really interesting and do, you know, work with my computer science team and let’s see what we come up with”. You can’t do that.",Data, B4,je,"Now they’re not giving you access to the data but they’re signposting you to the data. You still need to go through the right processes. You might need to go through the RegulatoryOrganisationName and get, err, you know, if you want patient identifiable information then there’s a whole confidentiality agreement process that has to happen and under certain circumstances they have to do that. Generically, what ends up happening is that a lot of vendors end up going to HealthOrganizationNameSubdivision and say “we have this technology, we want access to all your data” and they’re going to be, immediately, as soon as they go through any governance, like “No, we can’t do that” and that is a barrier",Data, B5,je,"projects within the development of this, we already had collected a bit of data for different elements of it. And unfortunately, one of those was from a housing association where we had had a faller who was actually wearing the technology at the time, and we’d gathered that data. We hadn’t done anything with it because at that moment in time what we were trying to demonstrate was, can we use this technology in a community, and gather data from it. Is it possible to do that. And yes it was. So we weren’t looking at algorithms at that time. We were just seeing, can we process the data from that community and get it in to ourselves, and collect it in a meaningful way. And yes we could. So what we had done is park that data",Data, B5,je,"we can share metrics within a cohort. You know, with their consent and start to gamify it",Data, B5,je,"a CountryName1 perspective, and the ethical requirement are a little different between CountryName1 and CountryName2 and that time, and because we weren’t going for the HealthOrganisationName we didn’t need to use the HealthOrganisatioName ethical approval process, which is entirely cumbersome. What we did what we did was we used the ethical processes of the individual councils that we were working with at the time. So the councils had to ethically approve it. Now fortunately we were working with some very smart, initially, some very smart people in the first council and they had their own ethical err process and a specific person that was managing that process, so we worked very closely with her to get that first ethical approval over the line. And so from there what we did was we identified the process we had to go through, and we did that. But on the data protection perspective, we make sure that everything is anonymised at source.",Data, B5,je,"gathered on I guess a commercial system, we use BrandName devices and so BrandName have an application on a phone that gathers the metrics directly from their wearable device and we use an API in the background to harvest the data from the BrandName system",Data, B5,je,"protection, the European data protection policy as a basic erm [pause] barrier to any kind of collection of data. Its not a barrier. You can do whatever you like if they fully understand what you’re doing and agree with it.",Data, B6,vu,its shapes and patterns that we’re trying to make predictions of,Data, B6,vu,we collect data from the built environment.,Data, B6,vu,its shapes and patterns that we’re trying to make predictions of,Data, B6,vu,"you have a lot of uncertainty in there as well so you have error in the sensors for example. So the temperature probes may be plus or minus one degree. So our stopping criteria, that really influences our stopping criteria because you can spend quite a lot of time messing about with what features we’re selecting and it turns out that all we’re doing is tuning for noise on a sensor",Data, D1,je,we need to get permission from them to get their data in a certain way,Data, D1,je,especially if you’re looking at sensitive data you’ve got to build that trust and that’s a harder conversation,Data, D1,je,"to be a lot more sensitive, personally identifiable data, the outcome, unless we are doing back office operations a lot of the, err, anything that’s going to have an impact on the clinical pathway that a patient would take or anything that would have an impact on whether someone’s going to get arrested or that. The level of trust, the level of certainty needs to be a lot higher and the sensitivity of the data’s a lot higher so we’re finding with those clients there’s an eagerness to explore stuff but they’re very conservative, quite rightly, I’d expect them to be conservative about the way their system is used.",Data, D1,je,"but a lot of people we work with, they don’t have it all centralised. They’re not optimised for use of that data",Data, D1,je,"if we need to sort out their data, if they’ve got duplicated data, if they’ve got no golden record, it might be that if we take a smaller project we can focus a bit more and do that.",Data, D1,je,"from the room people were saying that the dates in the system are generally not trustworthy. It’s the date that was entered when people enter hospital and its never revised so people go off their own gut instinct because they don’t trust the IT, or they don’t trust the data.",Data, D1,je,"just the data security aspect. When you’re dealing with sensitive data and you pass that around, how do you handle that. There has to be a lot of controls in place before, you know, you can even get the data. The other big thing I’ve seen certainly is just the lack of centralised data. Data is still very much siloed away. In many of the companies we work with anyway. We work with larger companies and naturally data grows in different areas. Its in different siloes and joining those together, because that’s where a lot of the power comes from, having these datasets joined together, is a pretty large task in itself often.",Data, D1,je,"I’d say data quality is pretty consistent between most people we work with, In that there are always issues",Data, D1,je,there will be quite a while before people are just more used to handling data and understanding how they need to start collecting and storing data to utilise these more advanced techniques,Data, D2,vu,"the understanding of data as an asset. So, you have your standard taxonomy based quality which is are things called the right thing. You have your availability of data concerns, you know have we got the right comparative granularity and those things. Through to how the data changes and is different and is difficult.  So, you know, how can it be true that the data looks like this when it didn’t yesterday?",Data, D2,vu,data governance as a conversation has evolved massively and the standard industry didn’t really keep up that well.,Data, D2,vu,"This comes out in the bias conversation and being able to just look at it, in ways that help you understand that bias is not usually built into the process. So if somebody says, oh you’ve built a great model, show me how that model applies to the genders, races, neurological spectrum disorders, different conditions, to people all those kind of things, and you get a blank look,",Data, D2,vu,"I don’t like that approach where you say data is everything if you record all the right data you don’t need anything, just look at the data. For me that’s ha, a model needs to be in there somewhere, so there’s some guidance. Even if the model is generated by a machine that’s generated in a different way you’ll still get to a model that’ll be evaluated somewhere.",Data, D3,je,"it’s a continual issue with our clients that data’s either not accessible which is a very kind of common theme. Organizations typically, at least the kind of organizations I’ve been working with, typically tend to have very disparate, a very disparate amount of data, spread across different locations with very little centralisation. So in fact this morning [laughs] I’ve had to drive to a client to pick up some data, Um so yes I’d say accessibility is a huge issue for us and that can block projects from starting. We’ve had a few clients where we’ve just been unable to start because we couldn’t get access to the data they have. Purely because its locked down and on an old system somewhere and is kind of inaccessible.",Data, D3,je,"The quality of data. People’s perception of what their data quality is doesn’t always line up. So we’ve had situations where we’ve um, we would kind of... we were given an account of what the data looks like, but through exploration and actually analysing this data we understood that it’s not quite as accurate as um people necessarily expect and therefore we weren’t going to get the outputs from applying machine learning to it that we wished to.",Data, D3,je,"but the accessibility of it, from an IT standpoint",Data, R1,vu,to effectively sell them data which had been analysed,Data, R1,vu,In practise the challenge in the past has been often scale. So its certainly interesting working with CompanyName that they’re doing live data science all the time. 5 years ago when we didn’t have cloud computing in the same way it was quite a challenge. Now its available but then the access is very carefully controlled and you have to be very careful about the security of it.,Data, R2,vu,"think about your data, what is data, what might you have or might you need, how might you store it or structure it, on the basis that if you solve those, if you do those things properly now you’re going to save yourself time later",Data, R2,vu,They were already dealing with big data in some sense and they were much more understanding in some sense of the needs for large amounts of data and large amounts of processing power,Data, R2,vu,their data plus external data about the environment and the demographics and weather and things like this,Data, R2,vu,"the outlier detection, the specific characteristic of the dataset was a lot of missing data. But then the company knew that because that was part of the challenge",Data, R2,vu,"you can’t just presume that here we have this data. They have this data, we can easily glue the two together",Data, R2,vu,And then maybe we’ll come back in 6 months a year to that client that’s then started collecting data and we can do some of the things they were originally interested in.,Data, R2,vu,"having to do the boring planning and how you structure your data, where you put it",Data, R3,je,"It took us well over 1 year of the 3 years project to actually get some real data from them. They were so concerned about getting the real data to us, even with nondisclosure and everything it took a long time to get to that stage",Data, R3,je,"[data sharing issues?] probably the first [customer confidentiality] more than the second [commercial sensitivity], but also probably this fear that the data might go public.",Data, R3,je,"even before the interview they created a small dataset – that’s for the interview, anonymised it, happy to release, because they actually want, for the task that we did at the interview the person to do it on the small subset of what we have. They don’t want us to invent synthetic data",Data, R4,vu,"all these standards, technologies, that sort of thing is knowledge representation",Data, R4,vu,"And none of that’s got anything to do with machine learning or anything like that, it’s just got to do with sharing information and the ability to query systems right? So that’s a much bigger problem, and only if you have that data available could you possibly start saying, now how could I optimise my 5000 boxes that are leaving PortName1 so that they all fit together in a minimum number of containers and arrive under the … you don’t have that [unclear] right?",Data, R4,vu,the ability to put it into the real world is all the time hitting practical issues like here we don’t have the data or the data is actually captured by an organisation in a different country and they won’t share it with us.,Data, R4,vu,"here in Europe we are concerned with personal privacy which in the US they are not but this is becoming more and more of an issue at every level, right? Is a company allowed to do anything with the data given that it was captured for purpose X are they really allowed to use it for purpose Y.",Data, R4,vu,"as soon as you get into the health, life science health world then people are very very worried about fulfilling their legal obligations, patient data and all that kind of thing. So that’s probably the biggest practical issue to get any kind of AI off the ground is are you prepared to share data with us for the purposes that might be interesting.",Data, R4,vu,"WebTranslationApp is based on hundreds of billions of pages of training data, right? You’ve got that problem yes? Have you got the training data? Have you got the mechanism to acquire the training data? Are you prepared to create a workflow, an interface, a platform, where people can, you know go through a series of steps, ask for specific services, and in the process also capture that data so you can subsequently improve your [unclear – process?]",Data, R4,vu,"this is still highly constrained by what data you are able to capture so at the moment we are getting very good at capturing weather data at a granular level because people are putting in weather stations in their fields, which would potentially allow you to do much more granular predictions as to when is the right moment to plant seeds to apply pesticides and things like that.",Data, B1,je,"So a confidence of 0.9, is that good? What is? And so your piece of work on the success measures of AI, people are really starting to think about that, thinking OK so 0.9 what? 0.9 things per thing? What’s the units of confidence [Interviewer: Success as a safety measure?] That’s exactly the question we’re trying to answer. How are we going to know.",SuccessCriteria, B1,je,"People when you say 1 in a million chance of dying today because of your driving or whatever it is, but you’ll go, yeah alright one in a million – I’ll do that. Then you say you’ve got a one in a million chance of being killed by an autonomous helicopter, yeah, I don’t think I fancy that very much.",SuccessCriteria, B1,je,"At the discussions and conferences and things I go to people are saying to me, oh yes the carpet cleaners have now learnt their reward scheme so they knock things over [pause] because their reward cycle is, I’ve got to have things to clean up. This floor is too clean, I get no reward here, but if I bash into the table, suddenly there’s a lot more mess for me to clean up, which is great! So as long as I go straight to the table first, hit the table and whatever’s on the table will fall off until I look around the table and Wow! There’s loads of stuff to clean! They are learning an unintended reward scheme to improve their reward score and people are thinking, well do we want a reward scheme then? OK let’s have another think about this. So that’s the level that we’re at. What do you really mean by reward score or confidence in? Um OK. ",SuccessCriteria, B1,je,"you get something called the Sorenson Dice Coefficient, I think that’s the guy's name, and it’s the number of false positives divided by the number of false negatives and it approaches 1 when you’ve got a really good separating techniques and its half when, well it could be that or that. And the closer it is to 1 the better the decision making process. It generally only works if you’ve got a binary decision to make",SuccessCriteria, B1,je,"one of your success criteria for a human is can you pass the driving test. You’re tested for one hour, in your home town, or nearby to your home town, with an oracle by the side of you that will look at how you behave on the road and that kind of thing. And they’ll say, yeah you’re passed and you can go off and drive on your own then. So if I’ve got my AI CompanyName car, or whatever through a driving test, and it passes the driving test are you happy for it to go on the road now?",SuccessCriteria, B1,je,"There’s a paper I was looking at where they’ve done it for wine tasting. So I said again, OK what here is your measure of success. So again, it’s the Turing test. We’re going to have seven wine tasters and an artificial intelligent taster. Alright, so how’s that going to work then? Well, we’re going to, first of all we’re going to train, one of the panel are all going to train this AI taster that when it tastes this type of red wine, it will first of all be able to identify what the red wine is and then say whether this is a good version of that red wine or not, dependent on the chemical constituents and all this kind of stuff and then recommend what menu choices you have with it. OK. And then we’’ put that in with another group of wine tasters and they’ll say, everyone will then agree whether its 91 out of a hundred for this type of wine or whatever, and then we’ll have the wine taster, our artificial intelligence wine taster give a score as well and then recommend, make a menu recommendation as well. Apparently, it worked, indistinguishable from the other wine tasters. Or gave a value that’s within the bounds of what you would have expected. Its not completely mad, it’s identifying the right wine and was giving it an appropriate score and was able to make a menu recommendation. They went right that’s our success, a comparison with a human oracle.",SuccessCriteria, B1,je,"in places where you do have an oracle it’s working. But that’s because I think you’ve got an oracle. So, the identification of breast cancer cells, I understand the current performance is as good as the radiographer in terms of the number of false positives and false negatives. But you had to have somebody who was a very good radiographer in the first place to train the AI. You needed that oracle to be in place to continually check and assess [interruption] … Yes so, the existence of an oracle or not and I would trust the radiographer, because they’re in a position of trust and they have a suitable qualified and experienced measure about them as to how or why they are an oracle, but would I trust another person on the street as a driver?",SuccessCriteria, B1,je,"I’ll trial and error to do things until I get a higher reward function. And then there’s lots of subtle bits about achieving a local minimum rather than the minimum, or a local maximum rather than the maximum. And then all the other odd things, nudging the table more to make things fall off it, so I get a better reward function.",SuccessCriteria, B2,je,"how do you determine what will be serendipitous to a user, what will be relevant to a user",SuccessCriteria, B2,je,"From our perspective it’s a large, very large, interest graph that’s able to… that has native, graph native machine learning model or simply that has graph learning and so its able to use the value of a highly connected graph to make highly accurate predictions that are ultimately recommendations. In our case highly trusted recommendations [Interviewer: the quality of recommendations is what your clients are buying?] Absolutely yes",SuccessCriteria, B2,je,"being able to expand the scope of participating merchants, either through online platforms like PlatformName1 where we’re integrated as well as extending our Neo4J community access to other graph data platforms where large strategic entities… It’s just basically making it easier for an enterprise to adopt our technology without having to go through too many integration hurdles.",SuccessCriteria, B2,je,or the end user to be able to access it easily it needs to be easily activated within their messaging applications,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted ,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B3,vu,Redacted,SuccessCriteria, B4,je,"yes Mr Multinational you have said your technology works in ophthalmology. Right, how do you scale that out? How does it work in local organizations? How do we provide a standard for that?",SuccessCriteria, B4,je,deploy solutions that will fundamentally save lives,SuccessCriteria, B4,je,which is what are the success factors of AI in the [vision?]. So technologically we can say this technology is AOK [unclear - to use?],SuccessCriteria, B4,je,how do we do this safely at scale,SuccessCriteria, B4,je,reducing the time for innovation to be adopted,SuccessCriteria, B4,je,the wider one is really how to fundamentally save lives,SuccessCriteria, B4,je,"I think that is about culture, and also about the user experience",SuccessCriteria, B5,je,"over-riding, I guess criterion, as all councils do, to demonstrate innovation. How they’re innovating and a lot of the metrics, so from the HealthSystemMonitoringOrganization, and all of those, they look at how you’re adopting new technologies and trying to become more efficient through those innovative steps",SuccessCriteria, B5,je,"if I run patientIdentifier through my new model will it identify the risks profiles around patientIdentifier, because that was a clear case, he actually fell over, where we had all of the data, and the answer was, yes we could.",SuccessCriteria, B5,je,"risk profile. So the first flag that we would have raised before patientIdentifier fell over would have been 24 days before the event. So we would have raised a risk flag, to tell the carers and to tell himself, you are now at risk of an adverse event, because we’ve identified significant inactivity within your daily life, and that, as it continues, is going to start to escalate your risk profile, and unfortunately create a scenario, an escalating risk scenario, that, as time progresses, you are much more likely to have some adverse outcome.",SuccessCriteria, B5,je,putting my metrics in with a known outcome and that known outcome was given to it by me because I know what good looks like and I know what [56:11] bad looks like.,SuccessCriteria, B5,je,"Innovation of the year, we’ve won wellbeing, I can’t remember all of them but we’ve won three major healthcare awards.",SuccessCriteria, B5,je,"Improving the health and welfare of their clients, and they can see most definitely this would do that. And also, and this is kind of contentious, but it’s one of cost reduction",SuccessCriteria, B6,vu,"Reliability, resilience and uptime",SuccessCriteria, D1,je,How you deploy it and you take into the fact the ethical considerations as well as the technical considerations,SuccessCriteria, D1,je,machine learning it evolves. How can you see whether it is still a good system,SuccessCriteria, D1,je,"we want to get an understanding of a business challenge and then understand the acceptance criteria on that. So number of transactions being processed, the time it takes, the accuracy, the quality",SuccessCriteria, D1,je,it really is a collaborative process even to understand what success might look like,SuccessCriteria, D1,je,"then there’s that change management so are the actual end users that are going to use this system, whether it be the end citizen or whether it be an employee, are they getting engaged. Are they buying in. Are they seeing the advantages. Are we being able to argue successfully against any fear factors or frustrations about this so it is a project that could scale. So I’d say we have these evolving success criterias through different phases of the project",SuccessCriteria, D2,vu,You know everything must be fair. What does it mean to be fair. How do you statistically judge whether something is fair or not.,SuccessCriteria, D2,vu,If you go and you say I want to make ten million pounds with my new data science department if you make ten million pounds you’ve been successful,SuccessCriteria, D3,je,the only definition of success is if we can at least improve on that problem and solve and improve that user’s experience,SuccessCriteria, D4,vu,the most tangible measure of success is was the continuation project [laughs] being funded!,SuccessCriteria, R1,vu,They weren’t expecting magic. They weren’t expecting you can put in any old variables and get a perfect answer. Coming out. They had a realistic expectation of what could be achieved.,SuccessCriteria, R1,vu,a real solution to a problem,SuccessCriteria, R1,vu,"its still not actual production, but its ready to go into production at the end. So they are taking some of what he does and they are using it as part of their natural practise day by day.",SuccessCriteria, R1,vu,"We’re quite good at working with people in terms of requirements elicitation. That’s crucial because that also affects the sorts of models you build. So you really need to understand what the cost functions are, what are the constraints we’re working under and so on. Otherwise you won’t be building a model that’ll be of any use to them.",SuccessCriteria, R1,vu,You have to be on the right path to what they want and not do it all and then find at the end its not what they needed.,SuccessCriteria, R1,vu,"Having good benchmarks is really important, because then you don’t over promise",SuccessCriteria, R1,vu,Many of the things I think are in operation,SuccessCriteria, R2,vu,the data didn’t seem to have the power to actually make those predictions,SuccessCriteria, R2,vu,"Potentially, they very much wanted to know if they could use their web analytics data and there was a very clear understanding. Again, there was a rich data source there and it could be potentially useful for all sorts of things",SuccessCriteria, R2,vu,all these successes are linked to the expectations you set at the beginning,SuccessCriteria, R2,vu,"my colleague works more on the optimisation that as I said has been very successful. So, we had, that tends to be more obvious success because if it can be measured, you can say I’m saving money on the bottom line because I know how to route my vans more efficiently",SuccessCriteria, R2,vu,There’s been some ongoing success that’s been linked into things like the clean air questions in CityName,SuccessCriteria, R3,je,the hub didn’t actually care because all they cared about is that the job gets done,SuccessCriteria, R3,je,"They were very results focussed and we could show them savings in terms of, or very clearly saying that. In the end it boiled down so to being able to predict numbers of pallets that they would have to expect at the end of the day, overall for the whole group as well as for individual member companies. And they were very satisfied with the numbers we had.",SuccessCriteria, R3,je,Also with the time that it took. So its both efficiency and quality of solutions. They didn’t care about how we did it and what was the AI element of it. All they cared about was to see whether this was better than what they had and it was,SuccessCriteria, R3,je,"after a few of these iterations it came out good enough, as it was, and there was absolutely no need to try and perfect it anymore",SuccessCriteria, R4,vu,"you have these interesting developments in image recognition or facial recognition and then you have these weird things like they just don’t operate on anyone who’s black, Asian minority, whatever",SuccessCriteria, R4,vu,"you’ve got police organizations around the world adopting face recognition technology, believing that it will work, even though the error levels in reality are at unacceptable levels",SuccessCriteria, R4,vu,The simple answer that you would like to be able to say is success will look like eventually a commercial product,SuccessCriteria, R4,vu,It may not be the magic you were expecting but magic will happen.,SuccessCriteria, B1,je,operated by artificial intelligence decision making process in an autonomous manner,RoleOfAI, B1,je,"after a year if you repeated the tests it might not pass it now, because it has learned a new goal structure, a new goal mechanism",RoleOfAI, B1,je,one of the practises we’re trying to have is an electromechanical monitor on an artificial intelligent technology – so you prove the monitor – no matter what the artificial intelligent bit does,RoleOfAI, B1,je,"We’re going to let the artificial intelligence evolve as it wants, but we’ll put a monitoring cage around it that we’re certain of.",RoleOfAI, B2,je,conversations initially between a user and a chatbot,RoleOfAI, B2,je,dynamically generating AI created catalogues which match the subject of an active conversation users are in and basically models keywords into attributes and phrases which are useful to rendering an update of the catalogue,RoleOfAI, B2,je,we’re generally introducing an AI based application as a secondary layer or secondary value add to what the primary application is,RoleOfAI, B2,je,an augmented reality interface that was embedded in the chatbot for social recommendations,RoleOfAI, B3,vu,Redacted,RoleOfAI, B4,je,"how a clinical technology, a clinical AI technology shouldn’t work on its own without any human interact…, any input",RoleOfAI, B4,je,"for example in radiology, we can have a second viewing by a machine, rather than a peer to peer human, and under exceptional circumstances if they don’t agree then it goes to another human, erm, how do we streamline things like that? That’s what I’m interested in",RoleOfAI, B5,je,"there’s trained and untrained. So we’re using a training model. So what I haven’t done is gone to … now classical AI – what people think about if they don’t think – AI is in some way going to sort all this out for me. Provide me with a solution, because AI’s that clever, and we’ll just throw all of my data into a black box, I’m going to turn the handle. Some neural network somewhere is going to figure all this out and provide me with an output. That never happens. That never ever happens.",RoleOfAI, B5,je,really its an advisory system,RoleOfAI, B6,vu,can you make a prediction of what the power consumption profile of that building,RoleOfAI, B6,vu,it moves that individual out of being overheads into being able to talk about the core business.,RoleOfAI, D1,je,AI should be decision support not making the decisions,RoleOfAI, D1,je,The AI will do the boring heavy lifting stuff you can do the more interesting work. And its finding the people within the organization that have got that ambition.,RoleOfAI, D1,je,we can give you an initial view on that and then a human then validates that AI view,RoleOfAI, D3,je,take away the repetitive parts of the job that machine learning and AI can learn to do and enable those employees to spend more time with customers,RoleOfAI, D3,je,if the technology’s abstracted and hidden. It kind of makes it easier to adopt that change,RoleOfAI, R1,vu,I did a project with the HealthOrganization in CityName to help them predict A&E admissions,RoleOfAI, R2,vu,they want to change their business,RoleOfAI, R2,vu,"they were, very much wanting to get into the smart city space and had quite a clear vision of how AI would fit into that",RoleOfAI, R2,vu,"Optimisation is a big thing, route planning",RoleOfAI, R3,je,"wasn’t quite AI but was still a very visual and technical solution. Sort of colour coded what they can expect where and traffic light system colour coding and yeah. [V data analysis system?] Yes data, prediction, demand prediction system. A lot of different little things but in terms of AI it was basically a prediction system. So if you like a regression machine learning it was a regression problem",RoleOfAI, R4,vu,what role can data play to have positive effects,RoleOfAI, R4,vu,"if you have a computational system where things are joined up you could reduce the time and complexity of getting these answers across federated actors who are only marginally sharing data, and really don’t want to share data. So, that’s how the big problem abut food and agriculture narrows down to very specific challenges that have very specific potential solutions: let’s create a data standard, let’s make sure everybody adopts something – JSON LD – so these systems can talk to each other",RoleOfAI, R4,vu,it’s even harder to explain to people what AI can potentially do for a company than blockchain,RoleOfAI, R4,vu,"One is logistics. There are lots and lots of problems in logistics and people think ‘well if we use machine learning we can optimise the um space used in lorries because they travel with 60% capacity most of the time and we will get all sorts of efficiency’. Actually, those are not the real problems in logistics if you actually dig down, they just want to know where things are. They want to know when something is arriving. That’s the big business problem.",RoleOfAI, R4,vu,"Here is theoretically an ideal scenario where you’d like to use artificial intelligence, because you could have a crop ripening model for the bananas. You should be able to capture data about when it was put into the fridge container in the Caribbean when it arrived in CountryName, when it was transferred into a ripening facility in CountryName. You should be able to have temperature records, IoT, all that kind of stuff and you should be able to say this is the moment it has to be pulled out of the ripening facility and sent to the supermarkets. It’s not happening, for all kinds of practical reasons right?",RoleOfAI, R4,vu,"So, company foundation documents and financial reports. These of course are produced in many different languages across the ContinentName  but whenever you have cross border interactions, whether its mergers, whether it’s some kind of thing the classic scenario is you go to your lawyer in CountryName1. CountryName1 lawyer tries to find a lawyer in CountryName2 or CountryName3 or somewhere and says ‘I need the documents for this company’ and they will go to chamber of commerce or wherever these things are, get them, then they have to be translated and all this. So this process is quite complex and cumbersome, and this start-up wants to automate this process using machine translation and things like that. But what’s interesting is although they have the idea ‘we need to simplify this process to make it easier, cheaper, more efficient, and we can use AI’",RoleOfAI, R4,vu,How could I push this into a digital layer and then do some kind of process on that where it’s getting more information about what’s going on or its optimising or predicting or something like that?,RoleOfAI, R4,vu,he’s created a start-up that’s going to do image recognition of strawberries so as to identify when strawberries are ripe or something like that,RoleOfAI,