Browse Source

Propose specification on Multi User User-Agent Extension (#326)

* Propose specification on Multi User User-Agent Extension

* Incorporate review feedback

* Drop intermediate format

* Clarify client contact information

* Clarify that client contact information is a URI

* Reduce internal cache mechanism visibility

* Add string TTL caching capabilities

* Conform to Multi User User-Agent Extension

This commit changes both the `User-Agent` header as specified for more
than one public follower and also follows the recommended token scheme.

See also: https://dev.twtxt.net/doc/useragentextension.html

* Fix typos

* Update assets

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
pull/376/head
Lyse 8 months ago
committed by GitHub
parent
commit
6e3f6cbc53
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
  1. 2
      CHANGELOG.md
  2. 103
      docs/_posts/2021-01-06-useragentextension.md
  3. 37
      internal/cache.go
  4. 640
      internal/rice-box.go
  5. 2
      internal/session/store.go
  6. 15
      internal/tokencache.go
  7. 52
      internal/ttlcache.go
  8. 12
      internal/utils.go
  9. 34
      internal/whofollows_handler.go

2
CHANGELOG.md

@ -191,7 +191,7 @@
* Fix profile template and profile type to show followers correctly with correct link
* Fix Profile.Type setting when calling .Profile() on models
* Fix a few misisng trimSuffix calls in some tempaltes
* Fix sessino persistence and increase default session timeout to 10days ([#49](https://github.com/prologic/twtxt/issues/49))
* Fix session persistence and increase default session timeout to 10days ([#49](https://github.com/prologic/twtxt/issues/49))
* Fix session unmarshalling caused by 150690c
* Fix the mess that is User/Feed URL vs. TwtURL ([#47](https://github.com/prologic/twtxt/issues/47))
* Fix user registration to disallow existing users and feeds

103
docs/_posts/2021-01-06-useragentextension.md

@ -0,0 +1,103 @@
---
layout: page
title: "Multi User User-Agent Extension"
category: doc
date: 2021-01-06 15:00:00
order: 3
---
At [twtxt.net](https://twtxt.net/) the **Multi User User-Agent** was invented
as an extension to the original [Twtxt Discoverability
Specification](https://twtxt.readthedocs.io/en/latest/user/discoverability.html).
## Purpose
Users can discover their followers if the followers include a specially
formatted `User-Agent` HTTP request header when fetching *twtxt.txt* files. The
original twtxt specification covers only single user clients. Since twtxt.net
is a multi user client, a single `GET` request is enough to present several
users the same feed. However, the `User-Agent` header needs to be modified when
several users on the same client instance are following a certain feed, so that
feed owners are still able to find out about their followers.
## Format
Depending on the number of followers on a multi user instance there are two
different formats to be used in the `User-Agent` HTTP request header.
### Single Follower
If there's only a single follower, the original twtxt specification on
[Discoverability](https://twtxt.readthedocs.io/en/latest/user/discoverability.html)
should be followed, to be backwards-compatible:
```
<client.name>/<client.version> (+<source.url>; @<source.nick>)
```
For example:
```
twtxt/1.2.3 (+https://example.com/twtxt.txt; @somebody)
```
### Multiple Followers
Starting with a second follower, the format changes. It aims to be fairly
compact:
```
<client.name>/<client.version> (~<who-follows.url>; contact=<client.contact-uri>)
```
For example:
```
twtxt/0.1.0@abcdefg (~https://example.com/whoFollows?token=randomtoken123; contact=https://example.com/support)
```
The feed URL and nick from the Single Follower format are replaced with just a
single Who Follows Resource URL, where all followers can be obtained. To aid
parsing and quickly differentiate these `User-Agent` headers from other
software, such as search engine spiders, the Who Follows URL is prefixed with a
tilde (`~`) rather than the plus sign (`+`).
An optional contact URL or e-mail address may be included as well. If present,
this should be either the client operator's e-mail address or a URL pointing to
a page were the client owner can be contacted.
### Who Follows Resource
When requested with the `Accept: application/json` header, this resource must
provide a JSON object with nicks as keys and their *twtxt.txt* file URLs as
values. The Format of the HTTP response body is:
```
{ "<nick>": "<url>" }
```
For example:
```
{
"somebody": "https://example.com/user/somebody/twtxt.txt",
"someoneelse": "https://example.com/user/someonelse/twtxt.txt"
}
```
## Security Considerations
Users of multi user clients should have the option to keep their following list
secret and thus to hide themselves from both the `User-Agent` as well as Who
Follows Resource.
The Who Follows Resource could be easily guessable and thus must be somehow
protected to not publicly disclose the followers of a certain feed to
unauthorized third parties. Keep in mind, the `User-Agent` header is only
available to the feed owner or web server operator. It must not be possible for
users, who see such a Who Follows Resource in their web server access logs, to
just swap out the own feed URL in a query parameter for a different feed and
get all the followers of that feed. The easiest way is to use a reasonably long
random token which internally is mapped to the feed URL and only valid for a
short period of time, e.g. one hour. The token should be rotated regularly.

37
internal/cache.go

@ -199,37 +199,22 @@ func (cache *Cache) FetchTwts(conf *Config, archive Archiver, feeds types.Feeds,
if followers != nil {
feedFollowers := followers[feed]
var userAgent string
if len(feedFollowers) == 1 {
headers.Set(
"User-Agent",
fmt.Sprintf(
"twtxt/%s (+%s; @%s)",
twtxt.FullVersion(),
URLForUser(conf, feedFollowers[0]), feedFollowers[0],
),
userAgent = fmt.Sprintf(
"twtxt/%s (+%s; @%s)",
twtxt.FullVersion(),
URLForUser(conf, feedFollowers[0]), feedFollowers[0],
)
} else {
var followersString string
if len(feedFollowers) > 5 {
followersString = fmt.Sprintf(
"%s and %d more... %s",
strings.Join(feedFollowers[:5], " "),
(len(feedFollowers) - 5), URLForWhoFollows(conf.BaseURL, feed),
)
} else {
followersString = strings.Join(feedFollowers, " ")
}
headers.Set(
"User-Agent",
fmt.Sprintf(
"twtxt/%s (Pod: %s Followers: %s Support: %s)",
twtxt.FullVersion(), conf.Name,
followersString, URLForPage(conf.BaseURL, "support"),
),
userAgent = fmt.Sprintf(
"twtxt/%s (~%s; contact=%s)",
twtxt.FullVersion(),
URLForWhoFollows(conf.BaseURL, feed, len(feedFollowers)),
URLForPage(conf.BaseURL, "support"),
)
}
headers.Set("User-Agent", userAgent)
}
cache.mu.RLock()

640
internal/rice-box.go

File diff suppressed because one or more lines are too long

2
internal/session/store.go

@ -11,7 +11,7 @@ import (
const DefaultSessionDuration = time.Hour
var (
ErrSessionNotFound = errors.New("sessin not found or expired")
ErrSessionNotFound = errors.New("session not found or expired")
ErrSessionExpired = errors.New("session expired")
)

15
internal/tokencache.go

@ -13,16 +13,13 @@ func init() {
tokenCache = NewTTLCache(1 * time.Hour)
}
func GenerateToken() string {
t := token.New()
ts := t.Encode()
func GenerateToken(feedurl string) string {
t := token.New().Encode()
for {
if tokenCache.Get(ts) == 0 {
tokenCache.Set(ts, 1)
return ts
if tokenCache.GetString(t) == "" {
tokenCache.SetString(t, feedurl)
return t
}
t = token.New()
ts = t.Encode()
t = token.New().Encode()
}
}

52
internal/ttlcache.go

@ -5,22 +5,22 @@ import (
"time"
)
type CachedItem struct {
Value int
Expiry time.Time
type cachedItem struct {
value interface{}
expiry time.Time
}
func (item CachedItem) Expired() bool {
return time.Now().After(item.Expiry)
func (item cachedItem) expired() bool {
return time.Now().After(item.expiry)
}
type CachedItems map[string]CachedItem
type cachedItems map[string]cachedItem
type TTLCache struct {
sync.RWMutex
ttl time.Duration
items map[string]CachedItem
items map[string]cachedItem
}
func (cache *TTLCache) Dec(k string) int {
@ -31,37 +31,63 @@ func (cache *TTLCache) Inc(k string) int {
return cache.Set(k, cache.Get(k)+1)
}
func (cache *TTLCache) Get(k string) int {
func (cache *TTLCache) get(k string) interface{} {
cache.RLock()
defer cache.RUnlock()
v, ok := cache.items[k]
if !ok {
return 0
}
return v.Value
return v.value
}
func (cache *TTLCache) Set(k string, v int) int {
func (cache *TTLCache) Get(k string) int {
v, ok := cache.get(k).(int)
if !ok {
return 0
}
return v
}
func (cache *TTLCache) GetString(k string) string {
v, ok := cache.get(k).(string)
if !ok {
return ""
}
return v
}
func (cache *TTLCache) set(k string, v interface{}) interface{} {
cache.Lock()
defer cache.Unlock()
cache.items[k] = CachedItem{v, time.Now().Add(cache.ttl)}
cache.items[k] = cachedItem{v, time.Now().Add(cache.ttl)}
return v
}
func (cache *TTLCache) Set(k string, v int) int {
val, _ := cache.set(k, v).(int)
return val
}
func (cache *TTLCache) SetString(k string, v string) string {
val, _ := cache.set(k, v).(string)
return val
}
func (cache *TTLCache) Reset(k string) int {
return cache.Set(k, 0)
}
func NewTTLCache(ttl time.Duration) *TTLCache {
cache := &TTLCache{ttl: ttl, items: make(CachedItems)}
cache := &TTLCache{ttl: ttl, items: make(cachedItems)}
go func() {
for range time.Tick(ttl) {
cache.Lock()
for k, v := range cache.items {
if v.Expired() {
if v.expired() {
delete(cache.items, k)
}
}

12
internal/utils.go

@ -1411,13 +1411,15 @@ func URLForTask(baseURL, uuid string) string {
)
}
func URLForWhoFollows(baseURL string, feed types.Feed) string {
token := GenerateToken()
func URLForWhoFollows(baseURL string, feed types.Feed, feedFollowers int) string {
return fmt.Sprintf(
"%s/whoFollows?uri=%s&nick=%s&token=%s",
"%s/whoFollows?followers=%d&token=%s",
strings.TrimSuffix(baseURL, "/"),
feed.URL, feed.Nick, token,
// Include the number of followers, so feed owners can use this as a vague
// indicator to avoid refetching our Who Follows Resource if the number did
// not change since they last checked their followers.
feedFollowers,
GenerateToken(feed.URL),
)
}

34
internal/whofollows_handler.go

@ -20,14 +20,11 @@ func (s *Server) WhoFollowsHandler() httprouter.Handle {
ctype = "json"
}
uri := r.URL.Query().Get("uri")
nick := r.URL.Query().Get("nick")
token := r.URL.Query().Get("token")
if uri == "" {
if token == "" {
if ctype == "html" {
ctx.Error = true
ctx.Message = "No URI supplied"
ctx.Message = "No token supplied"
s.render("error", w, ctx)
} else {
http.Error(w, "Bad Request", http.StatusBadRequest)
@ -35,19 +32,14 @@ func (s *Server) WhoFollowsHandler() httprouter.Handle {
return
}
if nick == "" {
log.Warn("no nick given to whoFollows request")
nick = "unknown"
}
if !ctx.Authenticated && tokenCache.Get(token) == 0 {
log.Warn("unauthenticated or invalid token for whoFollows request")
uri := tokenCache.GetString(token)
if uri == "" {
if ctype == "html" {
ctx.Error = true
ctx.Message = "You are not authorized to view this resource"
s.render("401", w, ctx)
ctx.Message = "Token expired or invalid"
s.render("error", w, ctx)
} else {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
http.Error(w, "Token Not Found", http.StatusNotFound)
}
return
}
@ -67,6 +59,7 @@ func (s *Server) WhoFollowsHandler() httprouter.Handle {
return
}
nick := ""
for _, user := range users {
if !user.IsFollowersPubliclyVisible && !ctx.User.Is(user.URL) {
continue
@ -74,8 +67,19 @@ func (s *Server) WhoFollowsHandler() httprouter.Handle {
if user.Follows(uri) {
followers[user.Username] = user.URL
if nick == "" {
for n, url := range user.Following {
if url == uri {
nick = n
break
}
}
}
}
}
if nick == "" {
nick = "unknown"
}
ctx.Profile = types.Profile{
Type: "External",

Loading…
Cancel
Save