In the past few years, the Internet of Things has started to become a reality; however, its growth has been hampered by privacy and security concerns. One promising approach is to use Semantic Web technologies to mitigate privacy concerns in an informed, flexible way. We present CARLTON, a framework for managing data privacy for entities in a Physical Web deployment using Semantic Web technologies. CARLTON uses context-sensitive privacy policies to protect privacy of organizational and personnel data. We provide use case scenarios where natural language queries for data are handled by the system, and show how privacy policies may be used to manage data privacy in such scenarios, based on an ontology of concepts that can be used as rule antecedents in customizable privacy policies.
Testing tools and AI - ideas what to try with some tool examples
Semantic Knowledge and Privacy in the Physical Web
1. Semantic Knowledge and
Privacy in
the Physical Web
PRAJIT KUMAR DAS, ABHAY KASHYAP,
GURPREET SINGH, CYNTHIA MATUSZEK,
TIM FININ, ANUPAM JOSHI
UMBC ebiquity and IRAL Labs
2. Motivation
Our goal is to provide contextually
aware information, using the IoT,
that is privacy preserving and
ubiquitously helpful
Image courtesy Batman Wikia
CARLTON
Slide 2 of 44
6. Salient features
CARLTON: A context-aware, NL question-answer BOT
Context derived from the Physical Web (IoT)
Under development, prototype system
Simple NLP using tools like Stanford CoreNLP
Mobile app and Kiosk for front-end
ABAC privacy model, Privacy rules using SWRL
Hierarchical context ontology
Optional authentication for UMBC people
Slide 6 of 44
7. Concretization of IoT
Small, quick seamless
interactions with
physical objects and
locations with your
device
Physical web: What?
Slide 7 of 44
18. “Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
Who is Tim
Finin?
System Overview
Slide 18 of 44
19. “Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
System Overview
Slide 19 of 44
20. “Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
Inference Engine
OntologyKnowledge base
System Overview
Slide 20 of 44
21. “Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generator
Context
Who is Tim
Finin?
Inference EngineResponse: JSON
{“text”: “He’s a
Professor in the
Computer Science
department!”}
OntologyKnowledge base
System Overview
Slide 21 of 44
24. User is a faculty and is in
front of Conf. room 1.
Is this room
booked from
2PM-3PM?
Example query
Slide 24 of 44
25. Conf. room 1 calendar has no
events during that time.
Is this room
booked from
2PM-3PM?
Example query
Slide 25 of 44
26. Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Example query
Slide 26 of 44
27. Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Yes, please!
Example query
Slide 27 of 44
28. Okay, the room
has been booked
in your name from
2PM – 3PM
Is this room
booked from
2PM-3PM? No, would you like
me to book it from
2PM – 3PM?
Yes, please!
Example query
Slide 28 of 44
43. Future work
Prototype system constantly adding conversations
Beacons on robots
Reason over robots near you
How robots respond to instructions?
“Can you take me to Prof. Matuszek now?”
“Show me the way to the ITE 346 conference
room”
Slide 43 of 44
44. Summary
We presented CARLTON
A context-aware, NL question-answer BOT
Context derived from the Physical Web (IoT)
Semantic web technologies used to preserve data privacy
Thanks to NSF for the travel grant!
and
Thanks to Google for the gift of beacons!
Slide 44 of 44
Hinweis der Redaktion
We present Carlton, a context-aware, natural language, question-answer BOT that responds to ‘Helpdesk’ styled questions using Semantic Web technologies
It leverages context derived from the Physical Web to protect privacy of organization’s and organizational entity’s data, by executing context-sensitive SWRL rules
Carlton is a conversational Bot that uses the Internet of Things to respond to queries tying in information about the environment
Roughly half of all consumers highly uncomfrotable with companies using and selling their data in physical spaces
Altimeter is part of Prophet, a consultancy company that helps clients find better ways to grow. Altimeter, is a research and consulting firm that helps companies understand – and act on – digital disruption.
Ratings of users in a range of 1-5, 1 being extremely uncomfortable and 2 being uncomfortable about the situation
Source: Consumer perceptions of the Internet of Things based on 2062 respondents
Carlton has a memory and remembers the question asked before the current one
Combining radio signals and an ultrasonic modem that uses the speakers and microphone of a mobile to determine that you are in close proximity to another device.
Everything is a tap away: Walk up and interact with any object -- a parking meter, a toy, a poster -- or location -- a bus stop, a museum, a store -- without installing an app first. Interactions are only a tap away.
PAIDPAY00:0000:1000:2000:3000:0000:1000:2000:30
A city rent-a-bike service could enable users to sign up on the spot
See what’s useful around you: See web pages associated with the space around you. Choose the page most useful to you.
SpotCall HumaneSociety
A dog collar could allow passerby to call a service to find the owner
Any object or place can broadcast content: When anything can offer information and utility, the possibilities are endless.
A bus that could alert users of its next stop
A home appliance could offer an interactive tutorial.
An industrial robot could display diagnostic information.
A mall that could offer a map.
Combining radio signals and an ultrasonic modem that uses the speakers and microphone of a mobile to determine that you are in close proximity to another device.
Everything is a tap away: Walk up and interact with any object -- a parking meter, a toy, a poster -- or location -- a bus stop, a museum, a store -- without installing an app first. Interactions are only a tap away.
PAIDPAY00:0000:1000:2000:3000:0000:1000:2000:30
A city rent-a-bike service could enable users to sign up on the spot
See what’s useful around you: See web pages associated with the space around you. Choose the page most useful to you.
SpotCall HumaneSociety
A dog collar could allow passerby to call a service to find the owner
Any object or place can broadcast content: When anything can offer information and utility, the possibilities are endless.
A bus that could alert users of its next stop
A home appliance could offer an interactive tutorial.
An industrial robot could display diagnostic information.
A mall that could offer a map.
Combining radio signals and an ultrasonic modem that uses the speakers and microphone of a mobile to determine that you are in close proximity to another device.
Everything is a tap away: Walk up and interact with any object -- a parking meter, a toy, a poster -- or location -- a bus stop, a museum, a store -- without installing an app first. Interactions are only a tap away.
PAIDPAY00:0000:1000:2000:3000:0000:1000:2000:30
A city rent-a-bike service could enable users to sign up on the spot
See what’s useful around you: See web pages associated with the space around you. Choose the page most useful to you.
SpotCall HumaneSociety
A dog collar could allow passerby to call a service to find the owner
Any object or place can broadcast content: When anything can offer information and utility, the possibilities are endless.
A bus that could alert users of its next stop
A home appliance could offer an interactive tutorial.
An industrial robot could display diagnostic information.
A mall that could offer a map.
Combining radio signals and an ultrasonic modem that uses the speakers and microphone of a mobile to determine that you are in close proximity to another device.
Everything is a tap away: Walk up and interact with any object -- a parking meter, a toy, a poster -- or location -- a bus stop, a museum, a store -- without installing an app first. Interactions are only a tap away.
PAIDPAY00:0000:1000:2000:3000:0000:1000:2000:30
A city rent-a-bike service could enable users to sign up on the spot
See what’s useful around you: See web pages associated with the space around you. Choose the page most useful to you.
SpotCall HumaneSociety
A dog collar could allow passerby to call a service to find the owner
Any object or place can broadcast content: When anything can offer information and utility, the possibilities are endless.
A bus that could alert users of its next stop
A home appliance could offer an interactive tutorial.
An industrial robot could display diagnostic information.
A mall that could offer a map.
Uses BLE and ultrasonic modem to get a URL to a phone!
Everything else is standard web technology or standard computing technology.
Nearby Messages: Provides Publish Subscribe methods relying on proximity
Nearby Connections: Enables local network real-time connect and exchange
Nearby Notifications: Android feature associating website or app to beacon
Uses BLE and ultrasonic modem to get a URL to a phone!
Everything else is standard web technology or standard computing technology.
Nearby Messages: Provides Publish Subscribe methods relying on proximity
Nearby Connections: Enables local network real-time connect and exchange
Nearby Notifications: Android feature associating website or app to beacon
Uses BLE and ultrasonic modem to get a URL to a phone!
Everything else is standard web technology or standard computing technology.
Nearby Messages: Provides Publish Subscribe methods relying on proximity
Nearby Connections: Enables local network real-time connect and exchange
Nearby Notifications: Android feature associating website or app to beacon
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
Nearby Messages API provides data from beacons to kiosks and Android phones
Stanford CoreNLP suite of tools used to do POS tagging, parsing and identify entities and relations
Context discovery of user and requester
antecedent ⇒ consequent
Antecedents: Requester context, Entity metadata, Entity context
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Faculty Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar No, do you want me to book it from 2PM – 3PM?
Student Is this room booked from 2PM-3PM? Conference room ITE Conference room 346 User id; Room calendar Please see CSEE office for booking
AppointmentRequest class addition
ValidReservationRequest as subclass of Restriction has_requestor => Faculty
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Advisee Is Dr. Joshi here? Dr. A. Joshi Department office User identity; Target disambiguation; Target location Dr. Joshi is in a meeting till 3PM
Student in class Is Dr. Joshi here? Dr. K. Joshi In front of faculty office User identity; Target disambiguation; Target location No, but I can tell you her office hours
User Query Target Requester Location Additional context Response
Visitor Where is Dr. Finin’s office? Dr. T. Finin ITE Building Target disambiguation; Please see ITE front desk for information
Student Where is Dr. Finin’s office? Dr. T. Finin ITE Building User Id; Target disambiguation; His office is in ITE 332
User Query Target Requester Location Additional context Response
Visitor Where is Dr. Finin’s office? Dr. T. Finin ITE Building Target disambiguation; Please see ITE front desk for information
Student Where is Dr. Finin’s office? Dr. T. Finin ITE Building User Id; Target disambiguation; His office is in ITE 332
User Query Target Requester Location Additional context Response
Visitor Where is Dr. Finin’s office? Dr. T. Finin ITE Building Target disambiguation; Please see ITE front desk for information
Student Where is Dr. Finin’s office? Dr. T. Finin ITE Building User Id; Target disambiguation; His office is in ITE 332
User Query Target Requester Location Additional context Response
Visitor Where is Dr. Finin’s office? Dr. T. Finin ITE Building Target disambiguation; Please see ITE front desk for information
Student Where is Dr. Finin’s office? Dr. T. Finin ITE Building User Id; Target disambiguation; His office is in ITE 332
User Query Target Requester Location Additional context Response
Visitor Where is Dr. Finin’s office? Dr. T. Finin ITE Building Target disambiguation; Please see ITE front desk for information
Student Where is Dr. Finin’s office? Dr. T. Finin ITE Building User Id; Target disambiguation; His office is in ITE 332
Complete privacy reasoning isomorphic to Truth maintenance
Truth maintenance - if information is being added in a forward manner how do you ensure that your knowledge already stored are still true -