IMAPSN client specification (draft v0.3)

1. Overview

The goal of IMAPSN is to create a social networking platform

that isn’t run by a single company
that provides a choice of revenue models (advertising, paid subscription)
where the users control their content and interactions
that is easy to set up
that a lot of people will use

There is already a platform in wide use that might fit the bill. Everyone already has email. There are existing choices in providers. People who care a little more about privacy can use paid services. Others can use advertising funded services. Implementing some of the social networking concepts on email will not require complicated deployment of anything new since the infrastructure already exists.

While an IMAPSN client could be implemented as a MUA (think thunderbird plugin), a web application platform and a browser interface is also a viable choice, and the goal of this specification is to define enough of the functionality so that a variety of clients can inter-operate. Then software developers can begin experimenting with creating their own user interfaces and approaches to attracting users. It’s important to note that, from the user perspective, IMAPSN is nothing like email. The user interface should look more like Facebook or Myspace than an email inbox.

Email as a storage and messaging platform has two serious drawbacks:

scaling into friend-of-friend territory will cause email traffic to grow very quickly and needs limits (namely comment visibility has to stay in the friend circle of the original message)
messages are pushed instead of pulled, which is a waste because not every status update from every friend will be read.

Assuming we can address these problems or live with some trade-offs, here is a design for implementing social networking concepts on email.

2. References

Reference Implementation: An in-progress java library being written to implement this specification.
Activity Streams Concepts and Representations: This document describes the activity construct and the object construct and the JSON representation of these.
activity-schema-01: This document describes the set of base object types and available verbs.
JSON schema for activities: full JSON schema for activities and object constructs checked into github.
http://json-schema.org/: general json schema info
http://code.google.com/apis/gmail/oauth/: info and examples on using oauth
Magic Signatures: draft spec for signing javascript objects

3. Definitions

account: a single email address or identity for which an IMAPSN client sends and receives messages. From the perspective of IMAPSN, all services (IMAP, SMPT) and their configuration information for a single email address form an account.
activity: “In its simplest form, an activity consists of an actor, a verb, and an object. It tells the story of a person performing an action on or with an object – ‘Geraldine posted a photo’ or ‘John shared a video’. In most cases these components will be explicitly declared, but they may also be implied.” – Activity Streams Concepts and Representations.
comment: The “comment” object type represents a textual response to another object.
friend: a reciprocal relationship between two users. user1 requests to make user2 a friend and user2 confirms. once the relationship is established the two users can send and receive activities as email messages.
group: a set of friends
status: The “status” Object type represents a human-readable update of the author’s situation, mood, location or other status.

4. Requirements

4.1 Functional Requirements

IMAPSN is built on an IMAP server and SMTP server.

An IMAPSN client to sends activities to friends using email. Activities received from friends are accessed on the IMAP server, processed, and displayed by the IMAPSN client.

The IMAPSN client will do the following:

Process a make friend request
Send a make friend request
Retrieve and display a list of friends
Display the details of a particular friend
Create groups (add and remove friends from groups)
Message an activity to a group (share status update, link, photo, make friend event, etc.)
Discard incoming messages that are not from a confirmed friend
Retrieve and display a list of activities
Allow the user to comment on and like activities
Allow the user to forward activities
Display the details of a particular activity
Message an incoming comment back out to the group to which the activity was originally sent
Add an account
Switch between accounts

4.2 Functional Flows

4.2.1 Use Case: make friend request and response

user1.client emails a friend-request message to user2
– time passes –
user2.client displays friend request to user2, user2 confirms
user2.client emails a friend-response message to user1 using the email address in the friend-request message.
user2.client emails an activity message with the make-friend verb to friends in the configured wall-group. (The activity only includes the new friend’s name, email address, and image.)
– time passes –
user1.client opens new friend-response message on user1.imap
user1.client emails an activity with the make-friend verb to wall-group.
user1.client notifies user1 of friendship.

4.2.2 Use Case: Message an activity to a group

User’s define their own groups. A special group called everybody includes all accpeted friends. This list correpsonds to all messages in the IMAPSN folder that have a subject that starts with “/contacts”.

user1 chooses a group to receive the message, and composes the message.
user1.client emails the news-item message to group
– time passes –
user2.client processes news-item message and displays in news list

4.2.3 Use case: Post to a wall

user1.client emails a wall-post message to a confirmed friend, user2.
– time passes –
user2.client processes new message containing wall-post.
user2.client emails wall-post message to configured wall-group.
– time passes –
userN.client (userN is a friend of user2) displays data from all wall-post messages received from user2 when userN chooses to view user2’s wall.

A variation of this is tagging multiple people in an image. The image is posted to the wall of each person tagged.

4.2.4 Use Case: Reply to Activities

This is the functional flow for user2 replying to (“liking” or “commenting” on) an activity received from user1.

user2 composes a reply or clicks “like” in response to an activity received from user1.
user2.client emails a comment message to user1.
– time passes –
user1.client processes new comment message.
user1.client emails comment message to all persons in the group the original message was emailed to.
– time passes –
userN.client (userN is a friend of user1 who received the original message) processes the incoming comment message, displaying it with the original.

By design, the replies are restricted to the persons included in the original message. Friends of user2 will not see user2’s replies made to user1, unless they are also friends of user1 and received the original message. user2 has the option of forwarding an activity with a comment. Then the replies will be limited to the group user2 forwarded to, which may or may not include user1.

4.2.5 Use Case: Forward an Activity

user2 receives news-item-1 from user1.
user2 wants to share news-item-1 with some of her own freinds
user2.client emails a forward message with an optional comment to a selected group. userN is a friend of user2 in the messaged group
– time passes –
userN.client processes new forward message and displays the forwarded news item in userN’s news stream.

It is possible for a user to have one of their items forwarded back to them. The forwarded item will collect it’s own set of replies separately from the original. User interfaces will have to convey that any comments on the forwarded item involve a different set of people making replies than the original and the people commenting on the forwarded item will not necessarily see the comments made on the original. The recommended approach is to make the forward a visually separate item in the news stream, with an indication of who forwarded the item.

4.2.6 Use Case: Update person data

user2 emails a person-update message containing user2’s person object to user1.
user1 updates person data for user2.

5. Interfaces

5.1 IMAP and SMTP

All data is accessed through IMAP APIs. See http://java.sun.com/products/javamail/javadocs/com/sun/mail/imap/IMAPStore.html or http://docs.python.org/library/imaplib.html.

Messages are sent using SMTP APIs. See http://java.sun.com/products/javamail/javadocs/com/sun/mail/smtp/package-summary.html or http://docs.python.org/library/smtplib.html

5.2 Data Format of E-mail messages

Each message will have a subject of “[IMAPSN] {message type} : {activity.title}”.

When addressing a message the To: address must match the From: address and all recipients must be included as BCC:.

Each message is a Multipart/alternative MIME email message that has the following parts:

application/json: the JSON representation of an activity serialized as a magic envelope.
text/plain: a user friendly plaintext representation
text/html: an optional user friendly html representation

An X-IMAPSN-object-id email header must be present and have as its value the id property of the encoded activity attached to the message.

5.3 Object References

It is often necessary to refer to an object for which the sender and receiver have the full representation. In this document the term object reference will be used to refer to an object in the activity schema that has the minimum required properties to identify the object. The two properties in an object reference are id and objectType.

5.4 Message Types

The following are message types sent and processed by an IMAPSN client. The data property of every magic envelope contains a single activity object. The following are the types of messages that an IMAPSN client must recognize.

5.4.1 `friend-request`

This is a message that includes data from a person making a request to be a friend. Message subject is “[IMAPSN] friend-request: {name of requestor}”. The attached JSON data is an activity with the following values:

id: a unique id for the activity

actor: `person` object of the one making the request.

verb: http://activitystrea.ms/schema/1.0/make-friend

object: A `person` object representing the requested friend. It
        will only include an `email` property, since the full data
        won't be known until after receiving a `friend-response`.

5.4.2 `friend-response`

This is a message that includes data from a person accepting a request to be a friend. Message subject is “[IMAPSN] friend-response: {name of responder}”. The attached JSON data is an activity with the following values:

id: a unique id for the activity

actor: The full person object of the one making the response.

verb: http://activitystrea.ms/schema/1.0/make-friend

object: An object reference to the `person` representing the
        friend who initiated the request.

inReplyTo: An object reference (containing just the id) to the
           friend-request activity.

5.4.3 `news-item`

These messages contain an attached activity object that is to be displayed in the news stream. The message subject is “[IMAPSN] news-item: {activity.title}”

5.4.4 `wall-post`

A wall-post is an activity with a person as a target. The message subject is “[IMAPSN] wall-post: {ativity.title}”.

The activity contains the following properties:

id: a unique id for the activity
actor: object reference to the `person` making the post
verb: any appropriate activity verb  (e.g., `tag`, `post`, `share`)
object: the object of the verb (e.g., a `photo`)
target: object reference to the `person` whose wall this is posted to

5.4.5 `direct-message`

This is a message addressed specifically to a friend or group. The mail subject is: “[IMAPSN] direct-message: {activity.title}”.

The activity contains the following properties:

id: a unique id for the activity
actor: object reference to the `person` sending the message
verb: post
object: any appropriate object, typically a `note`. Could also be a `photo`, etc.

5.4.6 `person-update`

This is a message that replaces the current data for a friend in the IMAPSN folder. The message subject is “[IMAPSN] person-update: {activity.title}” and it has an attached person object.

5.4.7 `comment`

The message subject is “[IMAPSN] comment: {activity.title}”. The activity has the following properties:

id: a unique id for the activity
actor: object reference to the `person` making the comment
inReplyTo: the object that the comment applies to. Only includes `id`.
verb: <http://activitystrea.ms/schema/1.0/post>
object: a relevant object. Typically a <http://activitystrea.ms/schema/1.0/comment>.

5.4.8 `forward`

A forward message has a subject of “[IMAPSN] forward: {activity.title}”.
Its properties are:

id: a unique id for the activity
actor: the person sharing the comment
verb: <http://activitystrea.ms/schema/1.0/reshare>
annotation: a comment about the object
object: the object that is being passed on (another activity).

This is similar to a “retweet” and gets processed as a news-item does.

6. Data Model

6.1 Data stored by the client

The following values are configured and stored locally by the client for each account the client connects to. The encryption mechanism used by the client to store passwords is not specified in this document.

field	type	description
account-name	string(255)	A name that is unique among accounts for users to identify this account.
displayName	string(255)	The full name of the user that is shown to other users--"John Doe".
email-id	string(255)	The email address used with this account.
imap-host	string(255)	The name of the IMAP host where there is IMAPSN data.
imap-port	integer	The port used when connecting to imap-host.
imap-user	string(100)	The username used to log into incoming-mail-server if oauth is not supported.
imap-password	string(100)	The password used to log into incoming-mail-server if oauth is not supported.
smtp-host	string(255)	The name of the SMTP host used to send IMAPSN messages.
smtp-port	integer	The port used when connecting to smtp-host.
smtp-user	string(100)	The username used to log into outgoing-mail-server if oauth is not supported.
smtp-password	string(100)	The password used to log into outgoing-mail-server if oauth is not supported.
private-key-password	string(100)	The password used to decrypt the user's private key. See Section 6.5.
imapsn-folder	string(30)	This is the folder on the IMAP server where all IMAPSN data is stored. Defaults to IMAPSN.

6.2 IMAP Directory layout

The data used by IMAPSN is stored in messages under an IMAP folder. Before storing any interface messages in IMAPSN folders the magic envelope is removed from the application/json attachment. All messages are stored in the configured “imapsn-folder” specified in the client’s account configuration. The default name for this folder is IMAPSN. Each message in the IMAPSN folder contains an attached JSON file. The subject of each message is a path where each part is separated by a ”/”. The final part of the path is the name of the attached file. Clients must not use nested IMAP Folders to organize messages since nested Folders are not supported by all IMAP servers.

The messages in the IMAPSN folder have the following subjects:

     + path: /contacts/*
     |
     + path: /inbox/*
     |
     + path: /news/*
     |
     + path: /wall/*
     |
     + path: /config/*
     |
     + message: /account-owner.json
     |
     + message: /person-groups.json
     |
     + message: /person-status-map.json
     |
     + message: /key-map.json

6.3 Format of Data Stored on IMAP Server But Never Messaged

Each message in the IMAPSN folder is a Multipart/alternative MIME email message that has the following parts:

application/json: the JSON representation serialized in plain text.
text/plain: an optional user friendly plaintext representation

6.4 IDs

The core object schema includes an id for every object. The convention for generating globally unique id’s will be to use acct:{email-id}#{UUID}. Where {email-id} is the account email addres and {UUID} is a universally unique identifier.

6.5 `/account-owner.json`

/account-owner.json is a the subject of a message stored in the IMAPSN folder. The application/json attachment of this message is an JSON object with one or more objects with the following properties.

field	type	description
account-owner	see [person][person.json]	the person data for this IMAPSN user. The data must have a "publicKey" property in the [magickey][application/magic-key] format, and a "keyhash" property that is the base64url-encoded SHA256 hash of the public signing key's magicsig representation. The person.id is formatted as: "acct:" + email_address + "#0"
privateKey	string	This is the encoded private key. See the next section for encoding format.
new-message-folder	string(30)	Name of IMAP folder where new IMAPSN messages are looked for. Defaults to root INBOX. By configuring this value users can set up a mail filtering rule that puts all new IMAPSN messages a user-specified folder.
wall-group	string (a group name)	The name of the group that is able to see messages posted to user's wall. Includes acceptance friend requests. Defaults to `everybody`.

6.5.1 Private Key Storage Format

First a string is constructed with two parts “PKCS8.{data}” where {data} is the private key serialized in PKCS#8 encoding and base64url encoded.

Next, the key is optionally encrypted. A key is derived from the “private-key-password” using PBKDF2WithHmacSHA1 and the private key string is encrypted using the derived key with the DES/CBC/PKCS5Padding encryption scheme. See http://tools.ietf.org/html/rfc2898.

Finally the three results of encryption are base64url encoded and serialized to a string with four parts separated by a ”.” (0x2E) character:

"DSA".<cipher text>.<salt>.<initialization vector>

The choice of whether to encrypt the key is left up to clients or users.

6.6 `/person-groups.json`

A message in the IMAPSN folder with the subject “/person-groups.json”. The application/json attachment of this message will contain a JSON object mapping each group-name property to an array of person references which contain the properties id, dislayName, and email. The email property is a string containing the email address. The id strings correspond to person objects found in /contacts messages.

6.7 `/person-status-map.json`

A message in the IMAPSN folder with the subject “/person-status-map.json”. The application/json attachment of this message will contain a JSON object mapping each person id to a status object with the folowing properties:

field	type	description
email	the address of this `person`
status	string	one of `pending`, `asleep`, `neglected`, or `active`.
last-sent	ISO 8601 date-time string	Time when an activity or message was last sent to this person.
last-received	ISO 8601 date-time string	Time when an activity or message was last received from this person.

6.8 `/key-map.json`

The key map is a message with the subject /key-map.json where the attached json data contains a map of “keyhash” values to public keys. The key hash is the base64url-encoded SHA256 hash of the public signing key’s magicsig representation. The public key is a string encoding of the pubic key in the magickey format.

6.9 `/contacts`

The IMAPSN folder will contain one message per person that is a confirmed friend.

The subject of each message will be “/contacts/{person.id}”.

The application/json attachment of this message is a file named “{person.id}” that contains a person object. Some of the important fields are listed here:

field	type	description
email	see [person][person.json]	This is where activities will be sent.
activity.title	see [core-object.json][]	This is what is displayed as text to represnt the person.
image	see [core-object.json][]	a media link construct referencing the avatar image displayed to represent the person.
publicKey	string	format is [magickey][application/magic-key]

6.9.1 media link construct

The info on the media link construct is sparse in current activity stream documentation. Here is an example of a media link construct:

{
    target: 'http://myvideos.com/raftingtrip/raftingvideo.avi',
    type: 'http://activitystrea.ms/schema/1.0/video',
    width: '400',
    height: '300',
    duration: '93' // in seconds
}

6.10 `/news`

Messages in the IMAPSN folder that are news-items will have a subject of “/news/{activity.id}”. The application/json attachment will be named {activity.id} and will contain an activity. Some of the important fields are listed here.

field	type	description
actor	object	this will be minimum data, that is, the id, since we have the full object.
verb	URI	For example `http://activitystrea.ms/schema/1.0/share`
object	object	The primary object of the activity. Example: a photo.

6.11 `/wall`

Messages in the IMAPSN folder that are wall-posts will have a subject of “/wall/{activity.id}”.

6.12 `/inbox`

Messages in the IMAPSN folder that are direct messages will have a subject of /inbox/{activity.id}”. The IMAP interface is used to mark them as read or not.

7. Program Control Flow

7.1 Client Start-up

These steps are taken every time the client starts up.

Process all new IMAPSN messages found in the “new-message-folder” conifgured in /account-owner.json.
Updates the status of each person in /person-status-map.json.
1. If status is pending the status is left alone.
2. if now - last-sent > 3 days set status to neglected.
3. or if now - last-received > 3 days set status to asleep.
4. otherwise set status to active.

7.2 Preprocess incoming email message

Before the processing for a specific message type, all incoming messages will usuall first be processed as follows:

Check the From: address of the email message. If it is not the address of a confirmed friend delete the message, otherwise continue.
Use the email address in the To: field of email message to look up the senders’s public key.
Verify the signature in the magic envelope. If the signature cannot be verified delete the email message, otherwise continue.
Look up the sender in the /person-status-map.json and update their status as prescribed in “Client Start-up”.

7.3 Sending and Processing Message Types

The following sections give more detailed steps for the messages described in the “Interfaces” section of this document.

7.3.1 Send `friend-request`

Construct the friend-request activity with a unique id, and make a signed magic envelope.
Email the friend-request message with the attached activity.
Make an initial entry in the /person-status-map.json.
1. id = activity.id
2. email = from friend request
3. status = pending
4. last-sent = now
5. last-received = null

The initial id value for the person-status-map entry is the the id of the friend request activity.

7.3.2 Process `friend-request`

Decode the incoming friend-request.
Verify signature using the public-key property from the person object of the friend request. If it doesn’t verify delete the message, otherwise save the keyhash and key in /key-map.json and continue.
Create an entry in the person status map:
1. id = from actor in friend-request activity
2. email = from actor in friend-request
3. status = active
4. last-sent = {current time} (when friend-response sent)
5. last-received = {current time} (when friend-request recieved)
Store a message containing the decoded person JSON object representing the new friend in the IMAPSN folder with a subject of “/contacts/{activity.actor.id}”.
Email a signed friend-response back to the requester. The activity.inReplyTo value must be the id of the incoming friend request.
Email a signed news-item with a http://activitystrea.ms/schema/1.0/make-friend verb to all friends in the configured wall-group.

7.3.3 Process a `friend-response`

Decode the incoming friend-response activity.
Verify signature using the public-key property from the person object of the friend response. If it doesn’t verify delete the message, otherwise continue.
Update person-status-map.json for the responding friend
1. Verify that a pending entry exists under the activity.inReplyTo id.
2. If it exists, remove the entry and continue, else, end.
3. make a new entry with the actiity.actor.id and set status = active
4. set last-received = {current time}
Store a message containing the person JSON object representing the new friend in the IMAPSN folder as /contacts/{person.id}.
Email a signed news-item with a http://activitystrea.ms/schema/1.0/make-friend to all friends in the configured wall-group.

7.3.4 Send `news-item`

Determine active group members in the specified group.
1. Look up each person in the group in /person-status-map.json.
2. If status is pending or asleep then do not include that person, otherwise include.
Email a signed news-item message BCC’ed to the active group members.
Update the last-sent time for each active group member that the message was sent to.
Place a copy of sent news-item in the IMAPSN folder with a subject of /news/{activity.id}. The recipients in the BCC: header will be used when processing incoming comment messages.

7.3.5 Process a `news-item`

Preprocess the incoming message.
Decode the news-item and place a copy of sent news-item in the IMAPSN folder with a subject of /news/{activity.id}.
The news item may now be displayed in the users’s news stream.

7.3.6 Process `wall-post`

Preprocess the incoming message.
Decode the wall-post
1. If the target of the activity is not the account owner place the wall post in the IMAPSN folder with a subject of /news/{activity.id}" and the UI should display the item in news stream. End.
2. If the target of the activity is the account owner, store the JSON activity in a message the IMAPSN folder with a subject of /wall/{activity.id}. Continue.
Determine the active group members of configured wall-group and email each a wall-post message with the activity attached. Update last-sent.
The wall post may now be displayed to the user when viewing the wall.

7.3.7 Send `comment`

Construct an activity and set inReplyTo to an object with an id that references the original.
Email a signed comment message to author of the original activity at the root of the inReplyTo chain. The root is the first object found when traversing inReplyTo properties that doesn’t have an inReplyTo property.

A comment may be made on other comments so the structure will be nested, and the email must be sent to the root person who initiated the discussion. This is important because the distribution of comments is handled by the original author.
Update the last-sent time for every person in the inReplyTo chain who is a friend.

7.3.8 Process `comment`

Preprocess the incoming message.
Decode the comment
Locate the original message that the comment was made inReplyTo in the IMAPSN folder with a subject of “/news/{value of inReplyTo}”.
If the original message was not created by account-owner the comment will be displayed by a UI in the news stream with the original. End.
Otherwise, if the original message was created by account-owner find list of recipients the original message was sent to.
Sign and email the comment to the original recipients.
Update the last-sent time for each recipient.
The comment may now be displayed to the user.

7.3.9 `forward` an activity

This is done just like a news-item except the activity should be constructed as described previously for a forward message.

Once an activity is forwarded, comments are sent to and distributed by the person who forwarded the message.

7.3.10 Process a `forward` activity

This is processed just like a news-item.

7.3.11 Send `direct-message`

Determine list of recipients (may be a group or may be directly adressed). Note: mesasges are sent regardless of status in person-status-map.json being pending or asleep.
Email a signed direct-message message to the recipients using the BCC: header.
Update the last-sent time in person-status-map.json for each recipient.
Place a copy of sent direct-message in the IMAPSN folder with a subject of “/inbox/{activity.id}”.

7.3.12 Process incoming `direct-message`

Preprocess the incoming direct-message.
Decode the direct-message and store the JSON activity in a direct-message within the IMAPSN folder with a subject of “/inbox/{activity.id}”.
The new message may now be displayed in the users’s message inbox.

IMAPSN client specification (draft v0.3)

1. Overview

2. References

3. Definitions

4. Requirements

4.1 Functional Requirements

4.2 Functional Flows

4.2.1 Use Case: make friend request and response

4.2.2 Use Case: Message an activity to a group

4.2.3 Use case: Post to a wall

4.2.4 Use Case: Reply to Activities

4.2.5 Use Case: Forward an Activity

4.2.6 Use Case: Update person data

5. Interfaces

5.1 IMAP and SMTP

5.2 Data Format of E-mail messages

5.3 Object References

5.4 Message Types

5.4.1 friend-request

5.4.2 friend-response

5.4.3 news-item

5.4.4 wall-post

5.4.5 direct-message

5.4.6 person-update

5.4.7 comment

5.4.8 forward

6. Data Model

6.1 Data stored by the client

6.2 IMAP Directory layout

6.3 Format of Data Stored on IMAP Server But Never Messaged

6.4 IDs

6.5 /account-owner.json

6.5.1 Private Key Storage Format

6.6 /person-groups.json

6.7 /person-status-map.json

6.8 /key-map.json

6.9 /contacts

6.9.1 media link construct

6.10 /news

6.11 /wall

6.12 /inbox